Sep 11 00:57:48.978756 ip-10-0-142-82 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 00:57:49.370384 ip-10-0-142-82 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 00:57:49.370384 ip-10-0-142-82 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 00:57:49.370384 ip-10-0-142-82 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 00:57:49.370384 ip-10-0-142-82 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 00:57:49.370384 ip-10-0-142-82 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 00:57:49.371025 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.370398    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 00:57:49.375220 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375204    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:49.375220 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375219    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375223    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375227    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375230    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375233    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375236    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375240    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375243    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375246    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375248    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375251    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375253    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375256    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375259    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375261    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375264    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375266    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375269    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375272    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375275    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:49.375283 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375280    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375285    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375288    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375291    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375294    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375297    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375300    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375303    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375306    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375309    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375311    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375314    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375317    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375319    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375322    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375325    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375327    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375330    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375332    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375335    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:49.375774 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375337    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375340    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375342    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375347    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375351    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375354    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375357    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375360    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375362    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375365    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375368    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375371    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375373    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375376    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375379    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375381    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375384    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375387    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375389    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375392    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:49.376268 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375394    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375397    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375399    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375402    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375405    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375408    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375410    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375413    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375415    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375418    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375420    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375423    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375425    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375428    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375430    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375433    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375437    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375442    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375458    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:49.376771 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375461    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375464    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375467    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375470    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375473    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375475    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375478    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375481    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375484    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375487    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375489    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375492    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375494    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375497    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375499    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375502    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375504    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375507    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:49.377238 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.375510    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377515    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377525    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377529    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377532    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377535    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377537    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377540    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377543    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377545    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377548    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377551    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377553    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377556    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377559    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377562    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377564    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377567    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377569    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:49.377692 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377572    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377574    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377577    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377580    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377582    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377585    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377587    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377589    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377592    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377594    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377597    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377600    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377602    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377605    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377607    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377611    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377613    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377616    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377618    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377621    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:49.378168 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377623    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377626    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377628    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377630    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377633    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377637    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377640    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377644    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377647    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377650    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377653    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377656    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377659    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377662    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377664    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377667    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377669    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377671    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377674    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377676    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:49.378854 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377679    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377681    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377684    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377686    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377689    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377691    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377694    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377699    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377702    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377706    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377709    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377712    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377715    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377717    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377720    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377722    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377725    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377727    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377730    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:49.379352 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377732    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377735    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377738    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377741    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377744    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377747    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377749    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377752    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377754    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377757    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377759    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377762    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377764    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377767    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377769    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377771    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377774    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377776    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377779    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377782    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:49.380004 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377785    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.377788    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378164    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378169    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378173    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378176    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378179    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378182    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378184    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378188    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378190    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378193    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378195    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378198    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378201    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378203    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378205    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378208    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378211    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378213    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:49.380686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378216    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378218    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378221    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378223    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378226    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378228    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378231    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378233    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378236    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378238    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378241    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378243    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378246    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378249    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378252    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378254    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378257    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378259    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378263    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378267    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:49.381206 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378270    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378273    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378276    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378278    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378282    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378285    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378288    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.378291    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379656    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379670    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379676    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379713    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379733    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379740    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379745    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379751    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379756    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379760    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379765    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379772    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:49.381711 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379776    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379781    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379786    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379797    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379801    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379809    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379816    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379823    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379828    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379833    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379838    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379843    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379848    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379853    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379860    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379863    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379865    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379870    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379873    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379876    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:49.382202 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379879    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379882    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379884    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379888    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379892    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379896    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379902    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379904    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379907    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379910    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379912    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379915    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379918    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379921    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379924    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379927    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379930    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379932    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379935    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:49.382726 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379940    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.379943    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380155    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380328    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380345    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380350    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380354    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380357    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380362    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380366    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380369    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380372    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380376    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380379    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380382    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380385    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380388    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380391    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380394    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380397    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380403    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380406    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380409    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380412    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 00:57:49.383183 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380415    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380419    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380422    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380425    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380428    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380432    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380435    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380438    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380441    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380444    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380461    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380465    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380468    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380471    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380474    2583 flags.go:64] FLAG: --enable-server="true"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380477    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380482    2583 flags.go:64] FLAG: --event-burst="100"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380485    2583 flags.go:64] FLAG: --event-qps="50"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380487    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380490    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380494    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380498    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380501    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380504    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380507    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 11 00:57:49.383802 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380510    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380512    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380516    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380518    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380521    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380524    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380527    2583 flags.go:64] FLAG: --feature-gates=""
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380531    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380534    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380537    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380541    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380544    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380547    2583 flags.go:64] FLAG: --help="false"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380550    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380553    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380556    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380559    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380563    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380567    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380570    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380573    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380575    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380578    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 00:57:49.384434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380581    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380584    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380587    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380591    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380593    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380597    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380600    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380602    2583 flags.go:64] FLAG: --lock-file=""
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380605    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380608    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380611    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380617    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380619    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380622    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380625    2583 flags.go:64] FLAG: --logging-format="text"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380628    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380631    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380634    2583 flags.go:64] FLAG: --manifest-url=""
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380637    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380642    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380645    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380649    2583 flags.go:64] FLAG: --max-pods="110"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380652    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380655    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380658    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 00:57:49.385081 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380661    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380664    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380667    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380670    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380678    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380681    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380684    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380687    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380690    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380696    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380699    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380702    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380705    2583 flags.go:64] FLAG: --port="10250"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380708    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380711    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d03bdbd7497b7d1"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380714    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380717    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380720    2583 flags.go:64] FLAG: --register-node="true"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380723    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380727    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380730    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380733    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380736    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380739    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380743    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 00:57:49.385720 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380751    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380756    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380759    2583 flags.go:64] FLAG: --runonce="false"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380762    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380765    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380769    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380771    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380774    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380777    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380785    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380789    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380791    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380794    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380797    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380800    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380803    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380806    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380809    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380815    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380818    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380821    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380826    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380828    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380831    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380834    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 00:57:49.386335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380837    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380840    2583 flags.go:64] FLAG: --v="2"
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380844    2583 flags.go:64] FLAG: --version="false"
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380848    2583 flags.go:64] FLAG: --vmodule=""
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380852    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.380855    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380969    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380973    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380976    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380980    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380983    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380985    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380988    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380991    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380993    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380996    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.380999    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381003    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381006    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381008    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:49.386961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381011    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381014    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381016    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381019    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381021    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381024    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381026    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381029    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381031    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381034    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381038    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381042    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381045    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381049    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381051    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381054    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381057    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381059    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381062    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381064    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:49.387510 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381067    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381070    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381072    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381075    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381078    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381080    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381083    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381086    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381088    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381091    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381094    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381097    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381100    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381102    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381105    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381108    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381111    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381113    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381116    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381118    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:49.387995 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381121    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381123    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381126    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381128    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381131    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381134    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381136    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381139    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381142    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381146    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381149    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381152    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381154    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381157    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381159    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381162    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381164    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381167    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381171    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:49.388575 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381175    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381178    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381180    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381183    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381187    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381194    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381197    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381200    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381203    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381206    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381209    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381212    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381215    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381218    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381221    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381224    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381226    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381232    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381234    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381237    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:49.389037 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381239    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381242    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381245    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381248    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381250    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.381253    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.381877    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.388435    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.388461    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.389322    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.389332    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389386    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389392    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389395    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389398    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:49.389639 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389401    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389405    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389409    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389412    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389415    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389418    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389421    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389423    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389426    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389428    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389432    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389437    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389440    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389442    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389458    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389460    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389463    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389466    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389469    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:49.390008 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389471    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389474    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389477    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389479    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389482    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389484    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389488    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389491    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389494    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389497    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389499    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389502    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389505    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389509    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389513    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389517    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389520    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389523    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389526    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389529    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:49.390465 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389531    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389534    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389537    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389540    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389542    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389545    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389547    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389550    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389553    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389555    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389558    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389560    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389563    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389566    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389569    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389571    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389574    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389576    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389580    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:49.391006 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389582    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389585    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389588    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389591    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389593    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389596    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389599    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389602    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389605    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389608    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389611    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389613    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389616    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389619    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389621    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389624    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389627    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389630    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389632    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389635    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:49.391471 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389637    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389640    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389643    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389646    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389649    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389651    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389654    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389657    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389660    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389662    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389665    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389668    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389671    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389674    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389678    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389681    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:49.391961 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389684    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.389689    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389797    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389802    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389805    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389809    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389812    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389815    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389818    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389820    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389823    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389826    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389828    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389831    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389834    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 00:57:49.392362 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389837    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389840    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389845    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389848    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389851    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389854    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389856    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389859    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389861    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389864    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389866    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389869    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389872    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389875    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389878    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389880    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389883    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389886    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389888    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 00:57:49.392738 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389891    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389893    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389896    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389898    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389901    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389904    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389906    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389909    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389912    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389914    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389918    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389920    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389923    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389925    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389929    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389932    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389935    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389938    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389941    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389944    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 00:57:49.393211 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389946    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389949    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389951    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389954    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389957    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389959    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389962    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389965    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389967    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389970    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389972    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389975    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389977    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389980    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389983    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389987    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389990    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.389993    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390000    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390003    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 00:57:49.393703 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390006    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390009    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390012    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390015    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390018    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390020    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390023    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390025    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390028    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390031    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390033    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390035    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390038    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390041    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390043    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390046    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390048    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390051    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390053    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390056    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 00:57:49.394187 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390059    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390061    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390064    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390066    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390069    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390071    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:49.390074    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.390079    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.390177    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.392132    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.393032    2583 server.go:1019] "Starting client certificate rotation"
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.393129    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 00:57:49.394668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.393159    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 00:57:49.395885 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.395872    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 00:57:49.400485 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.400470    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 00:57:49.412768 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.412739    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 11 00:57:49.417365 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.417349    2583 log.go:25] "Validated CRI v1 image API"
Sep 11 00:57:49.418087 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.418067    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 00:57:49.420432 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.420401    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7f268fcb-4611-4312-a091-371201a466b8:/dev/nvme0n1p3 f03e999a-4b72-4b7b-888b-84b117843ffb:/dev/nvme0n1p4]
Sep 11 00:57:49.420528 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.420431    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 00:57:49.422290 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.422269    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 00:57:49.425484 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.425359    2583 manager.go:216] Machine: {Timestamp:2026-09-11 00:57:49.424075399 +0000 UTC m=+0.332584882 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100170 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec295bd013e453099e510d84a794d777 SystemUUID:ec295bd0-13e4-5309-9e51-0d84a794d777 BootID:03047827-97fc-422b-9df7-997e0062ee99 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d8:c2:88:ee:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d8:c2:88:ee:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:f9:a3:c4:c9:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 00:57:49.425484 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.425476    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 00:57:49.425624 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.425574    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 00:57:49.428962 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.428938    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 00:57:49.429123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.428965    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 00:57:49.430329 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.430317    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 00:57:49.430385 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.430337    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 00:57:49.430385 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.430348    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 00:57:49.431166 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.431151    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 00:57:49.432074 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.432062    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 00:57:49.436187 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436173    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 00:57:49.436315 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436304    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 00:57:49.436369 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436341    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 00:57:49.436369 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436361    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 00:57:49.436476 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436376    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 00:57:49.436476 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436391    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 11 00:57:49.436476 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436405    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 00:57:49.436786 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436772    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 00:57:49.436854 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.436795    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 00:57:49.439138 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.439122    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 00:57:49.441632 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.441611    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 00:57:49.441698 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.441648    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 00:57:49.443092 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443080    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 00:57:49.443134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443099    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 00:57:49.443134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443107    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 00:57:49.443134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443113    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 00:57:49.443134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443128    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 00:57:49.443134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443134    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 00:57:49.443261 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443147    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 00:57:49.443261 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443156    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 00:57:49.443261 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443165    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 00:57:49.443261 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443170    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 00:57:49.443261 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443182    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 00:57:49.443261 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443187    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 00:57:49.443888 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443877    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 00:57:49.443919 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.443893    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 00:57:49.445905 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.445889    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 00:57:49.446066 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.446029    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 00:57:49.446139 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.446117    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 00:57:49.447763 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.447750    2583 server.go:1268] "Started kubelet"
Sep 11 00:57:49.447873 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.447838    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 00:57:49.447873 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.447837    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 00:57:49.448015 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.447903    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 00:57:49.448510 ip-10-0-142-82 systemd[1]: Started Kubernetes Kubelet.
Sep 11 00:57:49.450875 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.450122    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 00:57:49.450875 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.450194    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 00:57:49.453387 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.453359    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 00:57:49.456487 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.456467    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 00:57:49.457004 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.456991    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 00:57:49.457165 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.457122    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:49.457627 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.457606    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 00:57:49.457627 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.457615    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 00:57:49.457741 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.457637    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 00:57:49.457741 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.457708    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 00:57:49.457741 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.457719    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 00:57:49.458400 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.458384    2583 factory.go:55] Registering systemd factory
Sep 11 00:57:49.458400 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.458401    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 11 00:57:49.458621 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.458608    2583 factory.go:153] Registering CRI-O factory
Sep 11 00:57:49.458675 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.458625    2583 factory.go:223] Registration of the crio container factory successfully
Sep 11 00:57:49.458675 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.458666    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 00:57:49.458767 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.458681    2583 factory.go:103] Registering Raw factory
Sep 11 00:57:49.458767 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.458692    2583 manager.go:1196] Started watching for new ooms in manager
Sep 11 00:57:49.459044 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.459023    2583 manager.go:349] Starting recovery of all containers
Sep 11 00:57:49.463386 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.463359    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 00:57:49.463498 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.463420    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 00:57:49.464427 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.463326    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-82.ec2.internal.18d41e4d543ff899  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-82.ec2.internal,UID:ip-10-0-142-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-82.ec2.internal,},FirstTimestamp:2026-09-11 00:57:49.447723161 +0000 UTC m=+0.356232642,LastTimestamp:2026-09-11 00:57:49.447723161 +0000 UTC m=+0.356232642,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-82.ec2.internal,}"
Sep 11 00:57:49.468826 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.468806    2583 manager.go:354] Recovery completed
Sep 11 00:57:49.468913 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.468876    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-54tgx"
Sep 11 00:57:49.473023 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.473012    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:49.474436 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.474420    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-54tgx"
Sep 11 00:57:49.475157 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.475139    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:49.475236 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.475166    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:49.475236 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.475180    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:49.475809 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.475794    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 00:57:49.475809 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.475805    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 00:57:49.475957 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.475822    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 00:57:49.475957 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.475834    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:49.476552 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.476534    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:49.476640 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.476559    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:49.476640 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.476569    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:49.476963 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.476890    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-82.ec2.internal.18d41e4d55e28415  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-82.ec2.internal,UID:ip-10-0-142-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-82.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-82.ec2.internal,},FirstTimestamp:2026-09-11 00:57:49.475152917 +0000 UTC m=+0.383662396,LastTimestamp:2026-09-11 00:57:49.475152917 +0000 UTC m=+0.383662396,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-82.ec2.internal,}"
Sep 11 00:57:49.477937 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.477923    2583 policy_none.go:49] "None policy: Start"
Sep 11 00:57:49.478013 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.477942    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 00:57:49.478013 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.477955    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 00:57:49.480071 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.480049    2583 policy_none.go:47] "Start"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.515976    2583 manager.go:337] "Starting Device Plugin manager"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.516121    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.516133    2583 server.go:87] "Starting device plugin registration server"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.516310    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.516318    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.516420    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.516529    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.516538    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.516964    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.517003    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.617367    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.618344    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.618374    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.618387    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:49.620185 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.618412    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.627710 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.627695    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.627766 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.627715    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-82.ec2.internal\": node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:49.654409 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.654389    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:49.657182 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.657159    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 00:57:49.658340 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.658320    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 00:57:49.658416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.658369    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 00:57:49.658416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.658391    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 00:57:49.658522 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.658497    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 00:57:49.660548 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.660530    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:49.755023 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.755000    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:49.759286 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.759249    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal"]
Sep 11 00:57:49.759392 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.759326    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:49.760966 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.760948    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:49.761055 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.760986    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:49.761055 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.760997    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:49.763166 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.763152    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:49.763298 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.763282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.763354 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.763319    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:49.763974 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.763957    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:49.764064 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.763988    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:49.764064 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.764001    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:49.764064 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.764003    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:49.764064 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.764029    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:49.764064 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.764042    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:49.766279 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.766262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.766362 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.766294    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 00:57:49.767131 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.767108    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 00:57:49.767211 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.767137    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 00:57:49.767211 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.767150    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeHasSufficientPID"
Sep 11 00:57:49.783006 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.782977    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-82.ec2.internal\" not found" node="ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.786806 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.786789    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-82.ec2.internal\" not found" node="ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.855436 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.855411    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:49.859766 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.859751    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dfb5bbad91f5e0587cf031dfbabf92-config\") pod \"kube-apiserver-proxy-ip-10-0-142-82.ec2.internal\" (UID: \"f1dfb5bbad91f5e0587cf031dfbabf92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.859839 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.859773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b421af6133dcaa78c4b247180d4cc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal\" (UID: \"2b421af6133dcaa78c4b247180d4cc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.859839 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.859791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b421af6133dcaa78c4b247180d4cc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal\" (UID: \"2b421af6133dcaa78c4b247180d4cc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.955654 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:49.955591    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:49.959881 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.959866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dfb5bbad91f5e0587cf031dfbabf92-config\") pod \"kube-apiserver-proxy-ip-10-0-142-82.ec2.internal\" (UID: \"f1dfb5bbad91f5e0587cf031dfbabf92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.959929 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.959887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b421af6133dcaa78c4b247180d4cc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal\" (UID: \"2b421af6133dcaa78c4b247180d4cc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.959929 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.959904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b421af6133dcaa78c4b247180d4cc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal\" (UID: \"2b421af6133dcaa78c4b247180d4cc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.960033 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.959950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b421af6133dcaa78c4b247180d4cc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal\" (UID: \"2b421af6133dcaa78c4b247180d4cc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.960033 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.959961    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dfb5bbad91f5e0587cf031dfbabf92-config\") pod \"kube-apiserver-proxy-ip-10-0-142-82.ec2.internal\" (UID: \"f1dfb5bbad91f5e0587cf031dfbabf92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:49.960033 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:49.959984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b421af6133dcaa78c4b247180d4cc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal\" (UID: \"2b421af6133dcaa78c4b247180d4cc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:50.056078 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:50.056040    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:50.086516 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:50.086493    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:50.089662 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:50.089648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:51.528069 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:51.528042    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:51.628610 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:51.628537    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-82.ec2.internal\" not found"
Sep 11 00:57:51.717956 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.717928    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:51.729429 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.729408    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:51.741855 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.741837    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 00:57:51.743693 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.743678    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal"
Sep 11 00:57:51.751971 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.751955    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 00:57:51.765095 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.765077    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 00:57:51.765213 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.765196    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 00:57:51.765263 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.765245    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 00:57:51.809372 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.809354    2583 apiserver.go:52] "Watching apiserver"
Sep 11 00:57:51.824184 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.824168    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 00:57:51.824932 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.824912    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-fjfls","kube-system/konnectivity-agent-qqpsr","openshift-image-registry/node-ca-47kkt","openshift-multus/network-metrics-daemon-qrwkr","openshift-network-diagnostics/network-check-target-psmdm","openshift-network-operator/iptables-alerter-zdkvl","openshift-ovn-kubernetes/ovnkube-node-kwgdk","kube-system/global-pull-secret-syncer-m4sbv","kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx","openshift-cluster-node-tuning-operator/tuned-g8xmp","openshift-dns/node-resolver-k7zc7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal","openshift-multus/multus-additional-cni-plugins-6rb67"]
Sep 11 00:57:51.827343 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.827323    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.829069 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.829043    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 00:57:51.829543 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.829522    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 00:57:51.829629 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.829584    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-65s48\""
Sep 11 00:57:51.829629 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.829525    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 00:57:51.829629 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.829562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:57:51.829772 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.829612    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 00:57:51.830061 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.830041    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 00:57:51.832947 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.832335    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 00:57:51.832947 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.832522    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 00:57:51.832947 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.832615    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ztsqp\""
Sep 11 00:57:51.832947 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.832688    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.833274 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-multus-certs\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.833414 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e47f1012-5a54-441e-8fe1-f12d68b88038-konnectivity-ca\") pod \"konnectivity-agent-qqpsr\" (UID: \"e47f1012-5a54-441e-8fe1-f12d68b88038\") " pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:57:51.833508 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-system-cni-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.833616 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-socket-dir-parent\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.833674 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-cni-bin\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.833674 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-etc-kubernetes\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.833783 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-serviceca\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.833899 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzcnx\" (UniqueName: \"kubernetes.io/projected/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-kube-api-access-tzcnx\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.833961 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.833949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-cnibin\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834025 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45cbdc31-050e-4c41-baa8-0f184ca05be4-cni-binary-copy\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834072 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834059    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-kubelet\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834121 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-hostroot\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834168 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834134    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-conf-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834318 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5nc\" (UniqueName: \"kubernetes.io/projected/45cbdc31-050e-4c41-baa8-0f184ca05be4-kube-api-access-mf5nc\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834651 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-host\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.834721 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-cni-multus\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834721 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e47f1012-5a54-441e-8fe1-f12d68b88038-agent-certs\") pod \"konnectivity-agent-qqpsr\" (UID: \"e47f1012-5a54-441e-8fe1-f12d68b88038\") " pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:57:51.834721 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-cni-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834870 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-k8s-cni-cncf-io\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834870 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-netns\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.834870 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.834788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-daemon-config\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.835057 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.835037    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 00:57:51.835110 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.835093    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:51.835252 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:51.835221    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:57:51.835252 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.835248    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 00:57:51.835437 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.835357    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 00:57:51.835437 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.835416    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wwmd8\""
Sep 11 00:57:51.837118 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.837100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:51.837192 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:51.837158    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:57:51.839853 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.839838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:51.840184 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.840169    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 00:57:51.841929 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.841913    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:57:51.841992 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.841913    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 00:57:51.841992 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.841952    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 00:57:51.842070 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.842007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.842259 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.842244    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjflx\""
Sep 11 00:57:51.844203 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.844189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:51.844287 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:51.844247    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:57:51.844894 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.844877    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 00:57:51.844968 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.844919    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 00:57:51.844968 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.844954    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 00:57:51.845044 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.844979    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 00:57:51.845044 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.844981    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 00:57:51.845190 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.845172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8fv6z\""
Sep 11 00:57:51.845267 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.845176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 00:57:51.846689 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.846670    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:51.848959 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.848941    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vgk7n\""
Sep 11 00:57:51.849048 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.848965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 00:57:51.849048 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.848968    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 00:57:51.849048 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.848999    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 00:57:51.849048 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.849044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.850883 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.850837    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 00:52:50 +0000 UTC" deadline="2028-05-02 23:22:49.463433613 +0000 UTC"
Sep 11 00:57:51.850883 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.850882    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14398h24m57.612555285s"
Sep 11 00:57:51.851021 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.851009    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 00:57:51.851143 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.851128    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:57:51.851194 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.851164    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ch2rg\""
Sep 11 00:57:51.851494 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.851480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:51.853819 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.853802    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 00:57:51.853912 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.853855    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-td8k7\""
Sep 11 00:57:51.853912 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.853905    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 00:57:51.854094 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.854081    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:51.856116 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.856093    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 00:57:51.856178 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.856159    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 00:57:51.856380 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.856365    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l2lq6\""
Sep 11 00:57:51.926291 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.926268    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9tv7"
Sep 11 00:57:51.931225 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.931206    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 00:57:51.934132 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.934115    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9tv7"
Sep 11 00:57:51.935272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-cni-multus\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.935340 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-tuned\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.935340 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-run-netns\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.935415 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-socket-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:51.935415 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-cni-multus\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.935415 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-sys-fs\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:51.935500 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zbr\" (UniqueName: \"kubernetes.io/projected/ae4778e0-a283-452f-9ea0-f285a9c45750-kube-api-access-b5zbr\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:51.935500 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysctl-d\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.935500 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4778e0-a283-452f-9ea0-f285a9c45750-host-slash\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:51.935583 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45cbdc31-050e-4c41-baa8-0f184ca05be4-cni-binary-copy\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.935583 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-hostroot\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.935583 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-host\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.935668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-modprobe-d\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.935668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-lib-modules\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.935668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935638    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-host\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.935668 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935642    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-hostroot\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.935780 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-device-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:51.935780 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-system-cni-dir\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:51.935780 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-sys\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.935780 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:51.935780 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbjn\" (UniqueName: \"kubernetes.io/projected/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-kube-api-access-4cbjn\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:51.935918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:51.935918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:51.935918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-etc-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.935918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.935918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-env-overrides\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.935918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-k8s-cni-cncf-io\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-cnibin\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-systemd-units\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-k8s-cni-cncf-io\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.935993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovn-node-metrics-cert\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936009    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45cbdc31-050e-4c41-baa8-0f184ca05be4-cni-binary-copy\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdb4m\" (UniqueName: \"kubernetes.io/projected/ba79b33e-4621-4d88-8987-dbf54c5f5089-kube-api-access-rdb4m\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-socket-dir-parent\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-cni-bin\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-etc-kubernetes\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936134 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-serviceca\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-etc-kubernetes\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovnkube-config\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-socket-dir-parent\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-cni-bin\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovnkube-script-lib\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-var-lib-kubelet\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-registration-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936330    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2ng\" (UniqueName: \"kubernetes.io/projected/51b02402-9765-4d4c-a8eb-f7e817014c8e-kube-api-access-gf2ng\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-var-lib-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-node-log\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-cni-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-multus-certs\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-serviceca\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936488    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-multus-certs\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.936620 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-cni-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e47f1012-5a54-441e-8fe1-f12d68b88038-konnectivity-ca\") pod \"konnectivity-agent-qqpsr\" (UID: \"e47f1012-5a54-441e-8fe1-f12d68b88038\") " pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-kubernetes\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9227d14-ae58-49a3-9795-626c24256f78-dbus\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-system-cni-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-system-cni-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-conf-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5nc\" (UniqueName: \"kubernetes.io/projected/45cbdc31-050e-4c41-baa8-0f184ca05be4-kube-api-access-mf5nc\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-slash\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-conf-dir\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e47f1012-5a54-441e-8fe1-f12d68b88038-agent-certs\") pod \"konnectivity-agent-qqpsr\" (UID: \"e47f1012-5a54-441e-8fe1-f12d68b88038\") " pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/702a0279-0e7c-4718-9cd4-5eed0789e515-tmp\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-hosts-file\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24z5w\" (UniqueName: \"kubernetes.io/projected/2fca14ea-d910-47a0-9563-3ea2ff660cb6-kube-api-access-24z5w\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4778e0-a283-452f-9ea0-f285a9c45750-iptables-alerter-script\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-systemd\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.937465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-cni-bin\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-netns\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e47f1012-5a54-441e-8fe1-f12d68b88038-konnectivity-ca\") pod \"konnectivity-agent-qqpsr\" (UID: \"e47f1012-5a54-441e-8fe1-f12d68b88038\") " pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-daemon-config\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.936986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-run-netns\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysconfig\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-systemd\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-host\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-cni-netd\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937199    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzcnx\" (UniqueName: \"kubernetes.io/projected/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-kube-api-access-tzcnx\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysctl-conf\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4fw6\" (UniqueName: \"kubernetes.io/projected/702a0279-0e7c-4718-9cd4-5eed0789e515-kube-api-access-k4fw6\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-tmp-dir\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-498dg\" (UniqueName: \"kubernetes.io/projected/68c89915-cc67-481d-bf53-0a2656dd28bc-kube-api-access-498dg\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:51.938018 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-kubelet\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45cbdc31-050e-4c41-baa8-0f184ca05be4-multus-daemon-config\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9227d14-ae58-49a3-9795-626c24256f78-kubelet-config\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-cnibin\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-kubelet\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-cnibin\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-run\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cbdc31-050e-4c41-baa8-0f184ca05be4-host-var-lib-kubelet\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-ovn\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.938590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.937582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-log-socket\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:51.940010 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.939985    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e47f1012-5a54-441e-8fe1-f12d68b88038-agent-certs\") pod \"konnectivity-agent-qqpsr\" (UID: \"e47f1012-5a54-441e-8fe1-f12d68b88038\") " pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:57:51.945778 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.945760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzcnx\" (UniqueName: \"kubernetes.io/projected/9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35-kube-api-access-tzcnx\") pod \"node-ca-47kkt\" (UID: \"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35\") " pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:51.946287 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.946261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5nc\" (UniqueName: \"kubernetes.io/projected/45cbdc31-050e-4c41-baa8-0f184ca05be4-kube-api-access-mf5nc\") pod \"multus-fjfls\" (UID: \"45cbdc31-050e-4c41-baa8-0f184ca05be4\") " pod="openshift-multus/multus-fjfls"
Sep 11 00:57:51.965972 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:51.965955    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:52.021750 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.021725    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b421af6133dcaa78c4b247180d4cc26.slice/crio-67af0b2098c6ab75a532ecf9ef26daa65a586574c91fd80ec8cc86a20369aa45 WatchSource:0}: Error finding container 67af0b2098c6ab75a532ecf9ef26daa65a586574c91fd80ec8cc86a20369aa45: Status 404 returned error can't find the container with id 67af0b2098c6ab75a532ecf9ef26daa65a586574c91fd80ec8cc86a20369aa45
Sep 11 00:57:52.026363 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.026341    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dfb5bbad91f5e0587cf031dfbabf92.slice/crio-f8af0564ab4d757b455fab8be483e3808e93e0a1167b1205df4d4bba41150181 WatchSource:0}: Error finding container f8af0564ab4d757b455fab8be483e3808e93e0a1167b1205df4d4bba41150181: Status 404 returned error can't find the container with id f8af0564ab4d757b455fab8be483e3808e93e0a1167b1205df4d4bba41150181
Sep 11 00:57:52.027445 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.027430    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 00:57:52.034648 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.034613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal" event={"ID":"f1dfb5bbad91f5e0587cf031dfbabf92","Type":"ContainerStarted","Data":"f8af0564ab4d757b455fab8be483e3808e93e0a1167b1205df4d4bba41150181"}
Sep 11 00:57:52.035629 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.035609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal" event={"ID":"2b421af6133dcaa78c4b247180d4cc26","Type":"ContainerStarted","Data":"67af0b2098c6ab75a532ecf9ef26daa65a586574c91fd80ec8cc86a20369aa45"}
Sep 11 00:57:52.037828 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.037806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/702a0279-0e7c-4718-9cd4-5eed0789e515-tmp\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.037918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.037839    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-hosts-file\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:52.037918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.037866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24z5w\" (UniqueName: \"kubernetes.io/projected/2fca14ea-d910-47a0-9563-3ea2ff660cb6-kube-api-access-24z5w\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:52.037918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.037892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4778e0-a283-452f-9ea0-f285a9c45750-iptables-alerter-script\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:52.037918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.037914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-systemd\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.037936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-cni-bin\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.037946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-hosts-file\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.037962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysconfig\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-systemd\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysconfig\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-cni-bin\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-host\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-systemd\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-systemd\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.038123 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-host\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-cni-netd\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysctl-conf\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4fw6\" (UniqueName: \"kubernetes.io/projected/702a0279-0e7c-4718-9cd4-5eed0789e515-kube-api-access-k4fw6\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-cni-netd\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-tmp-dir\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.038316    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-498dg\" (UniqueName: \"kubernetes.io/projected/68c89915-cc67-481d-bf53-0a2656dd28bc-kube-api-access-498dg\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-kubelet\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9227d14-ae58-49a3-9795-626c24256f78-kubelet-config\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-run\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysctl-conf\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-ovn\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.038526    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret podName:a9227d14-ae58-49a3-9795-626c24256f78 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:52.538503005 +0000 UTC m=+2.075154191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret") pod "global-pull-secret-syncer-m4sbv" (UID: "a9227d14-ae58-49a3-9795-626c24256f78") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:52.038607 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-log-socket\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4778e0-a283-452f-9ea0-f285a9c45750-iptables-alerter-script\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-tuned\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-run\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-run-netns\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-tmp-dir\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038628    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-socket-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-ovn\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-sys-fs\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-run-netns\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9227d14-ae58-49a3-9795-626c24256f78-kubelet-config\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-kubelet\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zbr\" (UniqueName: \"kubernetes.io/projected/ae4778e0-a283-452f-9ea0-f285a9c45750-kube-api-access-b5zbr\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-socket-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-sys-fs\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.039426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-log-socket\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysctl-d\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4778e0-a283-452f-9ea0-f285a9c45750-host-slash\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-modprobe-d\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4778e0-a283-452f-9ea0-f285a9c45750-host-slash\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-sysctl-d\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-lib-modules\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-modprobe-d\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-device-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-system-cni-dir\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038985    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-lib-modules\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.038998    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-device-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-sys\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-system-cni-dir\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039087    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-sys\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbjn\" (UniqueName: \"kubernetes.io/projected/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-kube-api-access-4cbjn\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:52.040230 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-etc-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039221    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-etc-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.039228    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-env-overrides\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-cnibin\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-cnibin\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.039395    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs podName:2fca14ea-d910-47a0-9563-3ea2ff660cb6 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:52.539300844 +0000 UTC m=+2.075952037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs") pod "network-metrics-daemon-qrwkr" (UID: "2fca14ea-d910-47a0-9563-3ea2ff660cb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-systemd-units\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039457    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-systemd-units\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovn-node-metrics-cert\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-run-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.040923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdb4m\" (UniqueName: \"kubernetes.io/projected/ba79b33e-4621-4d88-8987-dbf54c5f5089-kube-api-access-rdb4m\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovnkube-config\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovnkube-script-lib\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-var-lib-kubelet\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039677    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-registration-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-var-lib-kubelet\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2ng\" (UniqueName: \"kubernetes.io/projected/51b02402-9765-4d4c-a8eb-f7e817014c8e-kube-api-access-gf2ng\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-var-lib-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-node-log\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51b02402-9765-4d4c-a8eb-f7e817014c8e-registration-dir\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-env-overrides\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-kubernetes\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9227d14-ae58-49a3-9795-626c24256f78-dbus\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-node-log\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041416 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68c89915-cc67-481d-bf53-0a2656dd28bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-slash\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.039975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040012    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-var-lib-openvswitch\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba79b33e-4621-4d88-8987-dbf54c5f5089-host-slash\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040055    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-kubernetes\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9227d14-ae58-49a3-9795-626c24256f78-dbus\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040300    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovnkube-script-lib\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/702a0279-0e7c-4718-9cd4-5eed0789e515-tmp\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovnkube-config\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68c89915-cc67-481d-bf53-0a2656dd28bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.040800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/702a0279-0e7c-4718-9cd4-5eed0789e515-etc-tuned\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.041897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.041709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba79b33e-4621-4d88-8987-dbf54c5f5089-ovn-node-metrics-cert\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.047102 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.047082    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:52.047195 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.047107    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:52.047195 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.047122    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6jfb9 for pod openshift-network-diagnostics/network-check-target-psmdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:52.047195 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.047182    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9 podName:ac1b18bb-5604-4428-8d7f-77edd1fb1806 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:52.54716676 +0000 UTC m=+2.083817939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6jfb9" (UniqueName: "kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9") pod "network-check-target-psmdm" (UID: "ac1b18bb-5604-4428-8d7f-77edd1fb1806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:52.047777 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.047753    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4fw6\" (UniqueName: \"kubernetes.io/projected/702a0279-0e7c-4718-9cd4-5eed0789e515-kube-api-access-k4fw6\") pod \"tuned-g8xmp\" (UID: \"702a0279-0e7c-4718-9cd4-5eed0789e515\") " pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.048609 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.048579    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zbr\" (UniqueName: \"kubernetes.io/projected/ae4778e0-a283-452f-9ea0-f285a9c45750-kube-api-access-b5zbr\") pod \"iptables-alerter-zdkvl\" (UID: \"ae4778e0-a283-452f-9ea0-f285a9c45750\") " pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:52.048759 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.048740    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2ng\" (UniqueName: \"kubernetes.io/projected/51b02402-9765-4d4c-a8eb-f7e817014c8e-kube-api-access-gf2ng\") pod \"aws-ebs-csi-driver-node-xzgqx\" (UID: \"51b02402-9765-4d4c-a8eb-f7e817014c8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.048832 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.048812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-498dg\" (UniqueName: \"kubernetes.io/projected/68c89915-cc67-481d-bf53-0a2656dd28bc-kube-api-access-498dg\") pod \"multus-additional-cni-plugins-6rb67\" (UID: \"68c89915-cc67-481d-bf53-0a2656dd28bc\") " pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.049435 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.049417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24z5w\" (UniqueName: \"kubernetes.io/projected/2fca14ea-d910-47a0-9563-3ea2ff660cb6-kube-api-access-24z5w\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:52.051138 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.051120    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbjn\" (UniqueName: \"kubernetes.io/projected/c41dccbf-cc64-4cbf-8d90-c270f6cd6f95-kube-api-access-4cbjn\") pod \"node-resolver-k7zc7\" (UID: \"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95\") " pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:52.051206 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.051197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdb4m\" (UniqueName: \"kubernetes.io/projected/ba79b33e-4621-4d88-8987-dbf54c5f5089-kube-api-access-rdb4m\") pod \"ovnkube-node-kwgdk\" (UID: \"ba79b33e-4621-4d88-8987-dbf54c5f5089\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.164073 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.163985    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjfls"
Sep 11 00:57:52.167878 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.167861    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:57:52.173967 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.173942    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cbdc31_050e_4c41_baa8_0f184ca05be4.slice/crio-bbf47ad4f75e5508f43d702a3ee5edf1415e2bf63ead2df8028a41ea28242cdd WatchSource:0}: Error finding container bbf47ad4f75e5508f43d702a3ee5edf1415e2bf63ead2df8028a41ea28242cdd: Status 404 returned error can't find the container with id bbf47ad4f75e5508f43d702a3ee5edf1415e2bf63ead2df8028a41ea28242cdd
Sep 11 00:57:52.174938 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.174918    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:52.176868 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.176841    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47f1012_5a54_441e_8fe1_f12d68b88038.slice/crio-3c3a4fb90276a7ea71adcecbe03cfd7dab2b1b2e32ca272a3062cd96f5420cbb WatchSource:0}: Error finding container 3c3a4fb90276a7ea71adcecbe03cfd7dab2b1b2e32ca272a3062cd96f5420cbb: Status 404 returned error can't find the container with id 3c3a4fb90276a7ea71adcecbe03cfd7dab2b1b2e32ca272a3062cd96f5420cbb
Sep 11 00:57:52.198555 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.198529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-47kkt"
Sep 11 00:57:52.207601 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.207584    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dda75f8_48cf_4a74_ba9a_e6df5c3b8e35.slice/crio-f4934b19cd972c4b52fadd85b8cb2bb3af4646d59df76a52b707498ee79ddf53 WatchSource:0}: Error finding container f4934b19cd972c4b52fadd85b8cb2bb3af4646d59df76a52b707498ee79ddf53: Status 404 returned error can't find the container with id f4934b19cd972c4b52fadd85b8cb2bb3af4646d59df76a52b707498ee79ddf53
Sep 11 00:57:52.219216 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.219198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdkvl"
Sep 11 00:57:52.224278 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.224264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:57:52.227792 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.227772    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4778e0_a283_452f_9ea0_f285a9c45750.slice/crio-9e705eb3e9d10ce789f612c87dbcc723b9b1401d34de8032225025f8ce606cc3 WatchSource:0}: Error finding container 9e705eb3e9d10ce789f612c87dbcc723b9b1401d34de8032225025f8ce606cc3: Status 404 returned error can't find the container with id 9e705eb3e9d10ce789f612c87dbcc723b9b1401d34de8032225025f8ce606cc3
Sep 11 00:57:52.231964 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.231947    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba79b33e_4621_4d88_8987_dbf54c5f5089.slice/crio-ca909985677c63978bb4ebcf8129a08d542a2da512840e997e7bbd73af849fa5 WatchSource:0}: Error finding container ca909985677c63978bb4ebcf8129a08d542a2da512840e997e7bbd73af849fa5: Status 404 returned error can't find the container with id ca909985677c63978bb4ebcf8129a08d542a2da512840e997e7bbd73af849fa5
Sep 11 00:57:52.244393 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.244376    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx"
Sep 11 00:57:52.252686 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.252669    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b02402_9765_4d4c_a8eb_f7e817014c8e.slice/crio-bf81f7fa06fc40849b41b9277a40a50e50b0be3a0508cc4614ebb1877ed759f3 WatchSource:0}: Error finding container bf81f7fa06fc40849b41b9277a40a50e50b0be3a0508cc4614ebb1877ed759f3: Status 404 returned error can't find the container with id bf81f7fa06fc40849b41b9277a40a50e50b0be3a0508cc4614ebb1877ed759f3
Sep 11 00:57:52.261115 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.261091    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8xmp"
Sep 11 00:57:52.268265 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.268247    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702a0279_0e7c_4718_9cd4_5eed0789e515.slice/crio-9f4455e8f8440de2dc8c4ff2329500c066f06fe154c0c361a48908cbb2f7e2fd WatchSource:0}: Error finding container 9f4455e8f8440de2dc8c4ff2329500c066f06fe154c0c361a48908cbb2f7e2fd: Status 404 returned error can't find the container with id 9f4455e8f8440de2dc8c4ff2329500c066f06fe154c0c361a48908cbb2f7e2fd
Sep 11 00:57:52.272261 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.272238    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k7zc7"
Sep 11 00:57:52.277353 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.277337    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6rb67"
Sep 11 00:57:52.280650 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.280633    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41dccbf_cc64_4cbf_8d90_c270f6cd6f95.slice/crio-5ec9bb628ceb97e620c8366055563ce12ae5a3793e334f1f461fde75295a28a1 WatchSource:0}: Error finding container 5ec9bb628ceb97e620c8366055563ce12ae5a3793e334f1f461fde75295a28a1: Status 404 returned error can't find the container with id 5ec9bb628ceb97e620c8366055563ce12ae5a3793e334f1f461fde75295a28a1
Sep 11 00:57:52.286870 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:57:52.286848    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c89915_cc67_481d_bf53_0a2656dd28bc.slice/crio-dcb97fb428325d4e333a203ba796bb43e1fcc8d66edd2a87390ebdbf0f29eeac WatchSource:0}: Error finding container dcb97fb428325d4e333a203ba796bb43e1fcc8d66edd2a87390ebdbf0f29eeac: Status 404 returned error can't find the container with id dcb97fb428325d4e333a203ba796bb43e1fcc8d66edd2a87390ebdbf0f29eeac
Sep 11 00:57:52.543898 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.543556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:52.543898 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.543645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:52.543898 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.543811    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:52.543898 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.543873    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret podName:a9227d14-ae58-49a3-9795-626c24256f78 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:53.54385175 +0000 UTC m=+3.080502941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret") pod "global-pull-secret-syncer-m4sbv" (UID: "a9227d14-ae58-49a3-9795-626c24256f78") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:52.544215 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.543999    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:52.544215 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.544065    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs podName:2fca14ea-d910-47a0-9563-3ea2ff660cb6 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:53.54404837 +0000 UTC m=+3.080699563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs") pod "network-metrics-daemon-qrwkr" (UID: "2fca14ea-d910-47a0-9563-3ea2ff660cb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:52.644406 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.644370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:52.644595 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.644521    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:52.644595 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.644539    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:52.644595 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.644551    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6jfb9 for pod openshift-network-diagnostics/network-check-target-psmdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:52.644788 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:52.644609    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9 podName:ac1b18bb-5604-4428-8d7f-77edd1fb1806 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:53.644585048 +0000 UTC m=+3.181236242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jfb9" (UniqueName: "kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9") pod "network-check-target-psmdm" (UID: "ac1b18bb-5604-4428-8d7f-77edd1fb1806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:52.937293 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.937188    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 00:52:51 +0000 UTC" deadline="2028-05-03 05:33:09.312966542 +0000 UTC"
Sep 11 00:57:52.937293 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:52.937248    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14404h35m16.375723143s"
Sep 11 00:57:53.033466 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.033435    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:53.033618 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.033571    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:57:53.054649 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.054611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"ca909985677c63978bb4ebcf8129a08d542a2da512840e997e7bbd73af849fa5"}
Sep 11 00:57:53.064128 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.064095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdkvl" event={"ID":"ae4778e0-a283-452f-9ea0-f285a9c45750","Type":"ContainerStarted","Data":"9e705eb3e9d10ce789f612c87dbcc723b9b1401d34de8032225025f8ce606cc3"}
Sep 11 00:57:53.077524 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.077495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-47kkt" event={"ID":"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35","Type":"ContainerStarted","Data":"f4934b19cd972c4b52fadd85b8cb2bb3af4646d59df76a52b707498ee79ddf53"}
Sep 11 00:57:53.092548 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.092520    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qqpsr" event={"ID":"e47f1012-5a54-441e-8fe1-f12d68b88038","Type":"ContainerStarted","Data":"3c3a4fb90276a7ea71adcecbe03cfd7dab2b1b2e32ca272a3062cd96f5420cbb"}
Sep 11 00:57:53.112877 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.112849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rb67" event={"ID":"68c89915-cc67-481d-bf53-0a2656dd28bc","Type":"ContainerStarted","Data":"dcb97fb428325d4e333a203ba796bb43e1fcc8d66edd2a87390ebdbf0f29eeac"}
Sep 11 00:57:53.120902 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.120881    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:53.126350 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.126327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k7zc7" event={"ID":"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95","Type":"ContainerStarted","Data":"5ec9bb628ceb97e620c8366055563ce12ae5a3793e334f1f461fde75295a28a1"}
Sep 11 00:57:53.136976 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.136951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx" event={"ID":"51b02402-9765-4d4c-a8eb-f7e817014c8e","Type":"ContainerStarted","Data":"bf81f7fa06fc40849b41b9277a40a50e50b0be3a0508cc4614ebb1877ed759f3"}
Sep 11 00:57:53.174837 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.174735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjfls" event={"ID":"45cbdc31-050e-4c41-baa8-0f184ca05be4","Type":"ContainerStarted","Data":"bbf47ad4f75e5508f43d702a3ee5edf1415e2bf63ead2df8028a41ea28242cdd"}
Sep 11 00:57:53.211433 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.211348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8xmp" event={"ID":"702a0279-0e7c-4718-9cd4-5eed0789e515","Type":"ContainerStarted","Data":"9f4455e8f8440de2dc8c4ff2329500c066f06fe154c0c361a48908cbb2f7e2fd"}
Sep 11 00:57:53.551790 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.551707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:53.551953 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.551790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:53.551953 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.551880    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:53.551953 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.551922    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:53.551953 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.551953    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret podName:a9227d14-ae58-49a3-9795-626c24256f78 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.55193423 +0000 UTC m=+5.088585423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret") pod "global-pull-secret-syncer-m4sbv" (UID: "a9227d14-ae58-49a3-9795-626c24256f78") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:53.552173 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.551973    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs podName:2fca14ea-d910-47a0-9563-3ea2ff660cb6 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.551963071 +0000 UTC m=+5.088614256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs") pod "network-metrics-daemon-qrwkr" (UID: "2fca14ea-d910-47a0-9563-3ea2ff660cb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:53.653040 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.652993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:53.653196 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.653173    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:53.653269 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.653197    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:53.653269 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.653211    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6jfb9 for pod openshift-network-diagnostics/network-check-target-psmdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:53.653392 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:53.653277    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9 podName:ac1b18bb-5604-4428-8d7f-77edd1fb1806 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.653258722 +0000 UTC m=+5.189909907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jfb9" (UniqueName: "kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9") pod "network-check-target-psmdm" (UID: "ac1b18bb-5604-4428-8d7f-77edd1fb1806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:53.938369 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.938263    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 00:52:51 +0000 UTC" deadline="2028-04-17 05:49:36.46058149 +0000 UTC"
Sep 11 00:57:53.938369 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:53.938321    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14020h51m42.522266056s"
Sep 11 00:57:54.031237 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:54.031177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:54.031428 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:54.031349    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:57:54.031772 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:54.031750    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:54.031870 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:54.031847    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:57:54.496912 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:54.496886    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 00:57:55.033500 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:55.033466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:55.033952 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.033593    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:57:55.567790 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:55.567757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:55.567968 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:55.567834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:55.568031 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.567967    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:55.568031 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.568026    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret podName:a9227d14-ae58-49a3-9795-626c24256f78 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:59.568008845 +0000 UTC m=+9.104660028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret") pod "global-pull-secret-syncer-m4sbv" (UID: "a9227d14-ae58-49a3-9795-626c24256f78") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:55.568144 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.568088    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:55.568144 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.568121    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs podName:2fca14ea-d910-47a0-9563-3ea2ff660cb6 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:59.568108431 +0000 UTC m=+9.104759611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs") pod "network-metrics-daemon-qrwkr" (UID: "2fca14ea-d910-47a0-9563-3ea2ff660cb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:55.668481 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:55.668334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:55.668688 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.668493    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:55.668688 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.668511    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:55.668688 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.668524    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6jfb9 for pod openshift-network-diagnostics/network-check-target-psmdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:55.668688 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:55.668579    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9 podName:ac1b18bb-5604-4428-8d7f-77edd1fb1806 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:59.668561532 +0000 UTC m=+9.205212719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jfb9" (UniqueName: "kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9") pod "network-check-target-psmdm" (UID: "ac1b18bb-5604-4428-8d7f-77edd1fb1806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:56.030897 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:56.030863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:56.031079 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:56.030863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:56.031079 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:56.030999    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:57:56.031191 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:56.031072    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:57:57.031732 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:57.031690    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:57.032158 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:57.031831    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:57:58.030708 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:58.030671    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:58.030887 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:58.030676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:58.030954 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:58.030800    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:57:58.030954 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:58.030919    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:57:59.031799 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:59.031768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:59.032248 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.031968    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:57:59.604864 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:59.604826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:57:59.605046 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:59.604896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:57:59.605046 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.604994    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:59.605046 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.605001    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:59.605260 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.605077    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret podName:a9227d14-ae58-49a3-9795-626c24256f78 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:07.605058843 +0000 UTC m=+17.141710038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret") pod "global-pull-secret-syncer-m4sbv" (UID: "a9227d14-ae58-49a3-9795-626c24256f78") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:57:59.605260 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.605099    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs podName:2fca14ea-d910-47a0-9563-3ea2ff660cb6 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:07.605087509 +0000 UTC m=+17.141738694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs") pod "network-metrics-daemon-qrwkr" (UID: "2fca14ea-d910-47a0-9563-3ea2ff660cb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:57:59.705581 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:57:59.705478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:57:59.705765 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.705648    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:57:59.705765 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.705669    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:57:59.705765 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.705680    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6jfb9 for pod openshift-network-diagnostics/network-check-target-psmdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:57:59.705765 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:57:59.705739    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9 podName:ac1b18bb-5604-4428-8d7f-77edd1fb1806 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:07.705722534 +0000 UTC m=+17.242373727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jfb9" (UniqueName: "kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9") pod "network-check-target-psmdm" (UID: "ac1b18bb-5604-4428-8d7f-77edd1fb1806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:00.031138 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:00.031064    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:00.031138 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:00.031115    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:00.031396 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:00.031204    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:00.031396 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:00.031339    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:01.031657 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:01.031619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:01.032111 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:01.031741    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:02.031511 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:02.031283    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:02.031688 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:02.031283    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:02.031688 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:02.031601    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:02.032057 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:02.031704    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:03.030995 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:03.030964    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:03.031172 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:03.031105    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:04.031054 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:04.030968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:04.031054 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:04.030981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:04.031575 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:04.031096    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:04.031575 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:04.031204    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:05.030942 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:05.030912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:05.031124 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:05.031071    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:06.030918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:06.030883    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:06.031121 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:06.030890    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:06.031121 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:06.030996    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:06.031121 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:06.031078    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:07.030777 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:07.030742    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:07.030953 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.030873    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:07.663744 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:07.663704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:07.664216 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:07.663796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:07.664216 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.663872    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:58:07.664216 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.663950    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret podName:a9227d14-ae58-49a3-9795-626c24256f78 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:23.663929499 +0000 UTC m=+33.200580687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret") pod "global-pull-secret-syncer-m4sbv" (UID: "a9227d14-ae58-49a3-9795-626c24256f78") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:58:07.664216 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.663877    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:58:07.664216 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.664041    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs podName:2fca14ea-d910-47a0-9563-3ea2ff660cb6 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:23.664016919 +0000 UTC m=+33.200668123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs") pod "network-metrics-daemon-qrwkr" (UID: "2fca14ea-d910-47a0-9563-3ea2ff660cb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:58:07.764561 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:07.764526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:07.764748 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.764695    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:58:07.764748 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.764724    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:58:07.764748 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.764737    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6jfb9 for pod openshift-network-diagnostics/network-check-target-psmdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:07.764883 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:07.764797    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9 podName:ac1b18bb-5604-4428-8d7f-77edd1fb1806 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:23.764784169 +0000 UTC m=+33.301435349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jfb9" (UniqueName: "kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9") pod "network-check-target-psmdm" (UID: "ac1b18bb-5604-4428-8d7f-77edd1fb1806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:08.031387 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:08.031317    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:08.031578 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:08.031317    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:08.031578 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:08.031427    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:08.031578 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:08.031505    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:09.030777 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:09.030738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:09.031220 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:09.030892    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:10.031324 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:10.031281    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:10.031798 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:10.031335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:10.031798 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:10.031425    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:10.031798 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:10.031718    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:11.031160 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:11.031122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:11.031266 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:11.031217    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:12.031575 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.031301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:12.032169 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.031371    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:12.032169 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:12.031651    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:12.032169 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:12.031741    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:12.257529 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.257501    2583 generic.go:356] "Generic (PLEG): container finished" podID="68c89915-cc67-481d-bf53-0a2656dd28bc" containerID="2c288b84831b7e511237c8992ccd658c48481384f0ee64fe1b6d1b3b927d18d3" exitCode=0
Sep 11 00:58:12.257675 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.257556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rb67" event={"ID":"68c89915-cc67-481d-bf53-0a2656dd28bc","Type":"ContainerDied","Data":"2c288b84831b7e511237c8992ccd658c48481384f0ee64fe1b6d1b3b927d18d3"}
Sep 11 00:58:12.259238 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.259156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k7zc7" event={"ID":"c41dccbf-cc64-4cbf-8d90-c270f6cd6f95","Type":"ContainerStarted","Data":"340ea4ca7e41042d2c4004883b6d2310c81ba69e6e684147b19afae63c2180ab"}
Sep 11 00:58:12.260736 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.260719    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx" event={"ID":"51b02402-9765-4d4c-a8eb-f7e817014c8e","Type":"ContainerStarted","Data":"f02689de1a070e1180bd931bffc59faa43fd9703129896156a9c64b1c5374019"}
Sep 11 00:58:12.262176 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.262156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjfls" event={"ID":"45cbdc31-050e-4c41-baa8-0f184ca05be4","Type":"ContainerStarted","Data":"2beb3602f8fe89b913721bffa5f393f295352711e167b9ca91a5bee9700e2ebb"}
Sep 11 00:58:12.263622 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.263603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal" event={"ID":"f1dfb5bbad91f5e0587cf031dfbabf92","Type":"ContainerStarted","Data":"199b81824233bd201e1f89b4f1efac201805c9f7daf65ad1286aef97839cd2e8"}
Sep 11 00:58:12.265080 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.265061    2583 generic.go:356] "Generic (PLEG): container finished" podID="2b421af6133dcaa78c4b247180d4cc26" containerID="2d4389c3e243e85aed5514df470f2ff8d6af944fa654b3d1341c0150c0c25524" exitCode=0
Sep 11 00:58:12.265157 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.265121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal" event={"ID":"2b421af6133dcaa78c4b247180d4cc26","Type":"ContainerDied","Data":"2d4389c3e243e85aed5514df470f2ff8d6af944fa654b3d1341c0150c0c25524"}
Sep 11 00:58:12.266448 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.266430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8xmp" event={"ID":"702a0279-0e7c-4718-9cd4-5eed0789e515","Type":"ContainerStarted","Data":"050b81bf9306708640e6084744db9bfd28114ff1d5cb389cd0621ead6cba68f9"}
Sep 11 00:58:12.268967 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.268950    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 00:58:12.269300 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.269282    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba79b33e-4621-4d88-8987-dbf54c5f5089" containerID="3ff5f3334221eca57d2711f5f628765017a6ff3129de65116b684fb570723a52" exitCode=1
Sep 11 00:58:12.269357 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.269334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"9a2d1e969c1afb634d675a9a046a980dba75d5c7bef894dddca991dec8cd19fc"}
Sep 11 00:58:12.269393 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.269384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"ec7a9e5e0be778e593e41e639ac95ccbb22c1192f5bda67affc220ede766d482"}
Sep 11 00:58:12.269428 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.269418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"650486ccaae1c9c0e0d854836b147d743db485be3a8c7c512b4275653f6f8f7a"}
Sep 11 00:58:12.269472 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.269454    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"e2ca6a76d229da86a94330d2e4d598c70439070cfd68a6bb526bd586476ef819"}
Sep 11 00:58:12.269527 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.269496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerDied","Data":"3ff5f3334221eca57d2711f5f628765017a6ff3129de65116b684fb570723a52"}
Sep 11 00:58:12.269572 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.269525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"9e8c28835bef16779f2715f2f84faa74113f366949d161f6034dbf77eb0b2e1e"}
Sep 11 00:58:12.270618 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.270601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-47kkt" event={"ID":"9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35","Type":"ContainerStarted","Data":"0cc447045e184ea0e08ef5fdf91bafd947223633f7ff89ba3b664b5762154a48"}
Sep 11 00:58:12.271932 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.271914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qqpsr" event={"ID":"e47f1012-5a54-441e-8fe1-f12d68b88038","Type":"ContainerStarted","Data":"5a33a2e24b41c86172fad5ef5fda25fb4d1552e1ba2c71be25132be21d097340"}
Sep 11 00:58:12.289912 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.289849    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qqpsr" podStartSLOduration=6.253500784 podStartE2EDuration="21.289836579s" totalImagesPullingTime="15.036335795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:12.289730676 +0000 UTC m=+21.826381876" watchObservedRunningTime="2026-09-11 00:58:12.289836579 +0000 UTC m=+21.826487780"
Sep 11 00:58:12.302485 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.302448    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-47kkt" podStartSLOduration=2.578790046 podStartE2EDuration="21.302436944s" totalImagesPullingTime="18.723646898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:12.302374076 +0000 UTC m=+21.839025300" watchObservedRunningTime="2026-09-11 00:58:12.302436944 +0000 UTC m=+21.839088170"
Sep 11 00:58:12.318704 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.318675    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g8xmp" podStartSLOduration=2.615130229 podStartE2EDuration="21.318664653s" totalImagesPullingTime="18.703534424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:12.318235574 +0000 UTC m=+21.854886776" watchObservedRunningTime="2026-09-11 00:58:12.318664653 +0000 UTC m=+21.855315853"
Sep 11 00:58:12.360066 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.360022    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k7zc7" podStartSLOduration=2.665214962 podStartE2EDuration="21.360004287s" totalImagesPullingTime="18.694789325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:12.340674455 +0000 UTC m=+21.877325666" watchObservedRunningTime="2026-09-11 00:58:12.360004287 +0000 UTC m=+21.896655490"
Sep 11 00:58:12.379839 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.379792    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-82.ec2.internal" podStartSLOduration=21.379781381 podStartE2EDuration="21.379781381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:12.359904162 +0000 UTC m=+21.896555363" watchObservedRunningTime="2026-09-11 00:58:12.379781381 +0000 UTC m=+21.916432582"
Sep 11 00:58:12.380212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.380186    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fjfls" podStartSLOduration=2.202112775 podStartE2EDuration="21.380178753s" totalImagesPullingTime="19.178065978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:12.379915635 +0000 UTC m=+21.916566826" watchObservedRunningTime="2026-09-11 00:58:12.380178753 +0000 UTC m=+21.916829957"
Sep 11 00:58:12.862789 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.862716    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 00:58:12.900678 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.900568    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T00:58:12.862730169Z","UUID":"1eee35ed-7eed-475c-8bf1-607308582fbf","Handler":null,"Name":"","Endpoint":""}
Sep 11 00:58:12.903452 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.903428    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 00:58:12.903550 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:12.903466    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 00:58:13.030986 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:13.030914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:13.031128 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:13.031026    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:13.276321 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:13.276269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx" event={"ID":"51b02402-9765-4d4c-a8eb-f7e817014c8e","Type":"ContainerStarted","Data":"897639571a0b85267d7d50ace707e523f5d71f729009488d4d7cc5ffc9367e8f"}
Sep 11 00:58:13.277982 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:13.277953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdkvl" event={"ID":"ae4778e0-a283-452f-9ea0-f285a9c45750","Type":"ContainerStarted","Data":"a3638ab4c885ce69f02fb3c5fc1b79633a66b416e7009675f043db60a85eecc2"}
Sep 11 00:58:13.292696 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:13.292650    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zdkvl" podStartSLOduration=3.589477827 podStartE2EDuration="22.292634238s" totalImagesPullingTime="18.703156411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:13.292339621 +0000 UTC m=+22.828990822" watchObservedRunningTime="2026-09-11 00:58:13.292634238 +0000 UTC m=+22.829285452"
Sep 11 00:58:13.851048 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:13.851011    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:58:13.851939 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:13.851916    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:58:14.030478 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:14.030455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:14.030577 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:14.030456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:14.030577 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:14.030567    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:14.030686 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:14.030623    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:14.281879 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:14.281852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal" event={"ID":"2b421af6133dcaa78c4b247180d4cc26","Type":"ContainerStarted","Data":"b02475ef55c0de1a1cd36c01acfdf67f363da34e4265400d95caf4ba36c0a6ec"}
Sep 11 00:58:14.285421 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:14.285399    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 00:58:14.286133 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:14.286105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"d69dd2c1e3f1473d93571c74243e047e3b2ec5b355cc325ecb6d280f2f312346"}
Sep 11 00:58:14.286603 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:14.286416    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:58:14.286957 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:14.286937    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qqpsr"
Sep 11 00:58:14.311765 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:14.311723    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-82.ec2.internal" podStartSLOduration=23.31170429 podStartE2EDuration="23.31170429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:14.297349403 +0000 UTC m=+23.834000605" watchObservedRunningTime="2026-09-11 00:58:14.31170429 +0000 UTC m=+23.848355493"
Sep 11 00:58:15.031506 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:15.031258    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:15.031795 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:15.031631    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:15.290942 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:15.290859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx" event={"ID":"51b02402-9765-4d4c-a8eb-f7e817014c8e","Type":"ContainerStarted","Data":"0f1aedb8bf40114a358b0e5669e36e1279ea9fa3eae85724c98e218c8befe695"}
Sep 11 00:58:15.311788 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:15.311742    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzgqx" podStartSLOduration=3.384184385 podStartE2EDuration="24.311728464s" totalImagesPullingTime="20.927544079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:15.31147501 +0000 UTC m=+24.848126204" watchObservedRunningTime="2026-09-11 00:58:15.311728464 +0000 UTC m=+24.848379662"
Sep 11 00:58:16.031176 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:16.031141    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:16.031405 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:16.031271    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:16.031405 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:16.031330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:16.031511 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:16.031411    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:17.031540 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:17.031462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:17.032041 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:17.031593    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:17.298806 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:17.298734    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 00:58:17.299122 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:17.299092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"bad44e8bfae545b1f7935d37d7f8cbd7a0f5039ffca376c0105fca1bb1544b34"}
Sep 11 00:58:17.299434 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:17.299417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:58:17.299570 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:17.299549    2583 scope.go:117] "RemoveContainer" containerID="3ff5f3334221eca57d2711f5f628765017a6ff3129de65116b684fb570723a52"
Sep 11 00:58:17.301223 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:17.301201    2583 generic.go:356] "Generic (PLEG): container finished" podID="68c89915-cc67-481d-bf53-0a2656dd28bc" containerID="fe21ed9b04ddf585a1e862f0de60fef8c4bd0568e777a8ef93f5dbb1b7713331" exitCode=0
Sep 11 00:58:17.301296 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:17.301249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rb67" event={"ID":"68c89915-cc67-481d-bf53-0a2656dd28bc","Type":"ContainerDied","Data":"fe21ed9b04ddf585a1e862f0de60fef8c4bd0568e777a8ef93f5dbb1b7713331"}
Sep 11 00:58:17.315846 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:17.315825    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:58:18.031489 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.031463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:18.031604 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.031488    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:18.032145 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:18.031595    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:18.032145 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:18.031716    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:18.306659 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.306629    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 00:58:18.307004 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.306981    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" event={"ID":"ba79b33e-4621-4d88-8987-dbf54c5f5089","Type":"ContainerStarted","Data":"1e153b89a4315480fa0664a91a8b3714e7b18d38033ea4ef7f95d9ab2ce35456"}
Sep 11 00:58:18.307270 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.307251    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:58:18.307409 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.307347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:58:18.309321 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.309281    2583 generic.go:356] "Generic (PLEG): container finished" podID="68c89915-cc67-481d-bf53-0a2656dd28bc" containerID="772db8c07c161437fd3f535535603d1ce81f9ddf9c3b10279bdae918330ff6d0" exitCode=0
Sep 11 00:58:18.309412 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.309334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rb67" event={"ID":"68c89915-cc67-481d-bf53-0a2656dd28bc","Type":"ContainerDied","Data":"772db8c07c161437fd3f535535603d1ce81f9ddf9c3b10279bdae918330ff6d0"}
Sep 11 00:58:18.323105 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.323088    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:58:18.360281 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.360233    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk" podStartSLOduration=8.608885556 podStartE2EDuration="27.360217079s" totalImagesPullingTime="18.751331523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:18.33374404 +0000 UTC m=+27.870395241" watchObservedRunningTime="2026-09-11 00:58:18.360217079 +0000 UTC m=+27.896868279"
Sep 11 00:58:18.382537 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.382510    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-psmdm"]
Sep 11 00:58:18.382665 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.382615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:18.382745 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:18.382722    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:18.385627 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.385602    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qrwkr"]
Sep 11 00:58:18.385729 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.385704    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:18.385833 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:18.385798    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:18.386426 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.386404    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m4sbv"]
Sep 11 00:58:18.386518 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:18.386486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:18.386599 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:18.386581    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:20.030502 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:20.030463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:20.031132 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:20.030512    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:20.031132 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:20.030474    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:20.031132 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:20.030586    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:20.031132 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:20.030678    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:20.031132 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:20.030738    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:20.316392 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:20.316365    2583 generic.go:356] "Generic (PLEG): container finished" podID="68c89915-cc67-481d-bf53-0a2656dd28bc" containerID="5b0a55a9434b66cb15df2ef6b733ba43360e1f779f1d225faed56b8a8b51629c" exitCode=0
Sep 11 00:58:20.316534 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:20.316450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rb67" event={"ID":"68c89915-cc67-481d-bf53-0a2656dd28bc","Type":"ContainerDied","Data":"5b0a55a9434b66cb15df2ef6b733ba43360e1f779f1d225faed56b8a8b51629c"}
Sep 11 00:58:22.030818 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:22.030626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:22.031248 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:22.030638    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:22.031248 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:22.030894    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:22.031248 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:22.030652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:22.031248 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:22.031005    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:22.031248 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:22.031057    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:23.680673 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:23.680632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:23.681234 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:23.680700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:23.681234 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:23.680770    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:58:23.681234 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:23.680801    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 00:58:23.681234 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:23.680847    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs podName:2fca14ea-d910-47a0-9563-3ea2ff660cb6 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:55.680824848 +0000 UTC m=+65.217476032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs") pod "network-metrics-daemon-qrwkr" (UID: "2fca14ea-d910-47a0-9563-3ea2ff660cb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 00:58:23.681234 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:23.680863    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret podName:a9227d14-ae58-49a3-9795-626c24256f78 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:55.680856574 +0000 UTC m=+65.217507754 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret") pod "global-pull-secret-syncer-m4sbv" (UID: "a9227d14-ae58-49a3-9795-626c24256f78") : object "kube-system"/"original-pull-secret" not registered
Sep 11 00:58:23.781628 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:23.781600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:23.781780 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:23.781742    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 00:58:23.781780 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:23.781756    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 00:58:23.781780 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:23.781764    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6jfb9 for pod openshift-network-diagnostics/network-check-target-psmdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:23.781880 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:23.781806    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9 podName:ac1b18bb-5604-4428-8d7f-77edd1fb1806 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:55.781795308 +0000 UTC m=+65.318446487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jfb9" (UniqueName: "kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9") pod "network-check-target-psmdm" (UID: "ac1b18bb-5604-4428-8d7f-77edd1fb1806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 00:58:24.031384 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.031296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:24.031545 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.031321    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:24.031545 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.031438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:24.031545 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.031445    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrwkr" podUID="2fca14ea-d910-47a0-9563-3ea2ff660cb6"
Sep 11 00:58:24.031545 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.031505    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-psmdm" podUID="ac1b18bb-5604-4428-8d7f-77edd1fb1806"
Sep 11 00:58:24.031738 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.031559    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m4sbv" podUID="a9227d14-ae58-49a3-9795-626c24256f78"
Sep 11 00:58:24.240640 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.240614    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-82.ec2.internal" event="NodeReady"
Sep 11 00:58:24.240796 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.240758    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 00:58:24.294963 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.294894    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-km5j6"]
Sep 11 00:58:24.331082 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.330955    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-657fb85584-4wbhr"]
Sep 11 00:58:24.331082 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.331059    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.335264 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.334892    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4r49p\""
Sep 11 00:58:24.337564 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.336110    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 00:58:24.337564 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.337243    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 00:58:24.339327 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.336157    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.339929 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.339910    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.342936 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.342915    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mrjfb"]
Sep 11 00:58:24.343531 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.343268    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.345067 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.345050    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 00:58:24.347656 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.347627    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 00:58:24.347755 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.347657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 00:58:24.347919 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.347893    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 00:58:24.348617 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.348589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hstjm\""
Sep 11 00:58:24.358082 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.358063    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 00:58:24.361032 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.360910    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd"]
Sep 11 00:58:24.361109 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.361044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.363685 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.363568    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.363878 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.363853    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 00:58:24.364118 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.364098    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bbj7f\""
Sep 11 00:58:24.364214 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.364143    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 00:58:24.365830 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.365808    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.371992 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.371947    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 00:58:24.380994 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.380975    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"]
Sep 11 00:58:24.381110 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.381092    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd"
Sep 11 00:58:24.384335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.384303    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4ld4h\""
Sep 11 00:58:24.384443 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.384409    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.384743 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.384723    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.401264 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.401245    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"]
Sep 11 00:58:24.401392 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.401374    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.403727 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.403708    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 00:58:24.403815 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.403718    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 00:58:24.403904 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.403879    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.403904 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.403882    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.404058 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.403961    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rr7ls\""
Sep 11 00:58:24.415169 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.415148    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"]
Sep 11 00:58:24.415274 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.415238    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.418757 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.418736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.418866 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.418741    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.418958 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.418935    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 00:58:24.419080 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.418962    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 00:58:24.419080 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.418991    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bpb4m\""
Sep 11 00:58:24.437292 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.437271    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6994df4f88-cwhzx"]
Sep 11 00:58:24.437423 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.437402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.439974 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.439924    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.439974 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.439939    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 00:58:24.439974 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.439941    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-q9lps\""
Sep 11 00:58:24.440245 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.440092    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 00:58:24.440347 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.440300    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.461341 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.461302    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"]
Sep 11 00:58:24.461435 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.461391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.464119 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.464099    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 00:58:24.464119 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.464121    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.464887 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.464369    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 00:58:24.464887 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.464763    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.465013 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.464985    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xsh95\""
Sep 11 00:58:24.465059 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.465023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 00:58:24.465059 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.465049    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 00:58:24.477500 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.477476    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-km5j6"]
Sep 11 00:58:24.477597 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.477510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:24.477597 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.477532    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd"]
Sep 11 00:58:24.477597 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.477554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"]
Sep 11 00:58:24.477597 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.477573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657fb85584-4wbhr"]
Sep 11 00:58:24.477597 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.477591    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"]
Sep 11 00:58:24.477838 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.477620    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"]
Sep 11 00:58:24.480580 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.480413    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 00:58:24.480580 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.480494    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.481065 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.481032    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.481161 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.481088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5gzc2\""
Sep 11 00:58:24.487519 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-image-registry-private-configuration\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.487619 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.487619 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-installation-pull-secrets\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.487619 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmtk\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-kube-api-access-vbmtk\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.487619 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/018f8615-58a3-4dc7-bc26-f5d740868701-trusted-ca\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.487798 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.487798 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a10e86f3-6bcd-42da-a580-e218495a161b-snapshots\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.487798 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27c3cca3-b650-4a88-a6c6-47a747ec800e-config\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.487798 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6qg\" (UniqueName: \"kubernetes.io/projected/a10e86f3-6bcd-42da-a580-e218495a161b-kube-api-access-hh6qg\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.487798 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/018f8615-58a3-4dc7-bc26-f5d740868701-serving-cert\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.487975 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a10e86f3-6bcd-42da-a580-e218495a161b-serving-cert\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.487975 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/191b3b23-de69-4658-a4ca-90bdef9cf7e1-ca-trust-extracted\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.487975 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-certificates\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.487975 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-bound-sa-token\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.487975 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/018f8615-58a3-4dc7-bc26-f5d740868701-config\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.487975 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4ws\" (UniqueName: \"kubernetes.io/projected/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-kube-api-access-bd4ws\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.487975 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwvw\" (UniqueName: \"kubernetes.io/projected/31932c40-b8a9-4ab3-8901-6d3d970aa7c5-kube-api-access-tfwvw\") pod \"volume-data-source-validator-86695b7cc7-qvzgd\" (UID: \"31932c40-b8a9-4ab3-8901-6d3d970aa7c5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd"
Sep 11 00:58:24.488272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.487990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-trusted-ca\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.488272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.488004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpxv\" (UniqueName: \"kubernetes.io/projected/27c3cca3-b650-4a88-a6c6-47a747ec800e-kube-api-access-rzpxv\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.488272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.488021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27c3cca3-b650-4a88-a6c6-47a747ec800e-serving-cert\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.488272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.488042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9glmv\" (UniqueName: \"kubernetes.io/projected/018f8615-58a3-4dc7-bc26-f5d740868701-kube-api-access-9glmv\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.488272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.488066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.488272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.488098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10e86f3-6bcd-42da-a580-e218495a161b-tmp\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.488272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.488134    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10e86f3-6bcd-42da-a580-e218495a161b-trusted-ca-bundle\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.488272 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.488165    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10e86f3-6bcd-42da-a580-e218495a161b-service-ca-bundle\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.490879 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.490856    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqpsr_e47f1012-5a54-441e-8fe1-f12d68b88038/konnectivity-agent/0.log" limit=1024
Sep 11 00:58:24.491577 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.491561    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mrjfb"]
Sep 11 00:58:24.491654 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.491590    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rncnx"]
Sep 11 00:58:24.491703 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.491662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.493927 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.493892    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 00:58:24.494112 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.494095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 00:58:24.494195 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.494136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.494283 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.494268    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.514339 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.514300    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"]
Sep 11 00:58:24.514454 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.514423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.516725 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.516706    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xx7m2\""
Sep 11 00:58:24.516811 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.516761    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 00:58:24.516811 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.516793    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 00:58:24.532529 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.532509    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"]
Sep 11 00:58:24.532651 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.532633    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"
Sep 11 00:58:24.535448 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.535426    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-n25sr\""
Sep 11 00:58:24.535537 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.535426    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 00:58:24.546789 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.546736    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b"]
Sep 11 00:58:24.546899 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.546882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.549863 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.549847    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 00:58:24.549968 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.549901    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 00:58:24.550044 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.550026    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 00:58:24.550104 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.550023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 00:58:24.566745 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.566726    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-drvkn"]
Sep 11 00:58:24.566865 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.566851    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b"
Sep 11 00:58:24.569371 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.569353    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.569483 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.569385    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.569653 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.569522    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mzdzs\""
Sep 11 00:58:24.578717 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578699    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6994df4f88-cwhzx"]
Sep 11 00:58:24.578808 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578731    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"]
Sep 11 00:58:24.578808 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578753    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rncnx"]
Sep 11 00:58:24.578808 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578778    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b"]
Sep 11 00:58:24.578808 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578797    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"]
Sep 11 00:58:24.578988 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578815    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drvkn"]
Sep 11 00:58:24.578988 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578833    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"]
Sep 11 00:58:24.578988 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578859    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"]
Sep 11 00:58:24.578988 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.578877    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"]
Sep 11 00:58:24.579168 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.579005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drvkn"
Sep 11 00:58:24.581480 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.581459    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vpvqt\""
Sep 11 00:58:24.581685 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.581666    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 00:58:24.581765 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.581688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 00:58:24.581765 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.581717    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 00:58:24.588547 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588501    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfcx\" (UniqueName: \"kubernetes.io/projected/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-kube-api-access-dhfcx\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.588547 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588555    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqpsr_e47f1012-5a54-441e-8fe1-f12d68b88038/konnectivity-agent/0.log" limit=1024
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/191b3b23-de69-4658-a4ca-90bdef9cf7e1-ca-trust-extracted\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-certificates\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-bound-sa-token\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-stats-auth\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4ws\" (UniqueName: \"kubernetes.io/projected/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-kube-api-access-bd4ws\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwvw\" (UniqueName: \"kubernetes.io/projected/31932c40-b8a9-4ab3-8901-6d3d970aa7c5-kube-api-access-tfwvw\") pod \"volume-data-source-validator-86695b7cc7-qvzgd\" (UID: \"31932c40-b8a9-4ab3-8901-6d3d970aa7c5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd"
Sep 11 00:58:24.588734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cefebf34-9896-4f29-84b3-596e6bbb0b9e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2nnm\" (UniqueName: \"kubernetes.io/projected/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-kube-api-access-s2nnm\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-trusted-ca\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9glmv\" (UniqueName: \"kubernetes.io/projected/018f8615-58a3-4dc7-bc26-f5d740868701-kube-api-access-9glmv\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-default-certificate\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10e86f3-6bcd-42da-a580-e218495a161b-service-ca-bundle\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9kg\" (UniqueName: \"kubernetes.io/projected/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-kube-api-access-dg9kg\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.588994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-config-volume\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-installation-pull-secrets\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-image-registry-private-configuration\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.589108 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmtk\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-kube-api-access-vbmtk\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a10e86f3-6bcd-42da-a580-e218495a161b-snapshots\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589142    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hljh\" (UniqueName: \"kubernetes.io/projected/79c84350-cf67-43ba-9abb-dd3a881108d0-kube-api-access-6hljh\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/018f8615-58a3-4dc7-bc26-f5d740868701-serving-cert\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a10e86f3-6bcd-42da-a580-e218495a161b-serving-cert\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-certificates\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/79c84350-cf67-43ba-9abb-dd3a881108d0-klusterlet-config\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10e86f3-6bcd-42da-a580-e218495a161b-tmp\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10e86f3-6bcd-42da-a580-e218495a161b-trusted-ca-bundle\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpxv\" (UniqueName: \"kubernetes.io/projected/27c3cca3-b650-4a88-a6c6-47a747ec800e-kube-api-access-rzpxv\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27c3cca3-b650-4a88-a6c6-47a747ec800e-serving-cert\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/018f8615-58a3-4dc7-bc26-f5d740868701-config\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-tmp-dir\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10e86f3-6bcd-42da-a580-e218495a161b-service-ca-bundle\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.589568 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szbc\" (UniqueName: \"kubernetes.io/projected/cefebf34-9896-4f29-84b3-596e6bbb0b9e-kube-api-access-4szbc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79c84350-cf67-43ba-9abb-dd3a881108d0-tmp\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27c3cca3-b650-4a88-a6c6-47a747ec800e-config\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/018f8615-58a3-4dc7-bc26-f5d740868701-trusted-ca\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6qg\" (UniqueName: \"kubernetes.io/projected/a10e86f3-6bcd-42da-a580-e218495a161b-kube-api-access-hh6qg\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cefebf34-9896-4f29-84b3-596e6bbb0b9e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a10e86f3-6bcd-42da-a580-e218495a161b-snapshots\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-trusted-ca\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.590212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.589783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.590624 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.590250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10e86f3-6bcd-42da-a580-e218495a161b-tmp\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.590624 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.590569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10e86f3-6bcd-42da-a580-e218495a161b-trusted-ca-bundle\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.590624 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.590580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/191b3b23-de69-4658-a4ca-90bdef9cf7e1-ca-trust-extracted\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.590724 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.590672    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:24.590724 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.590687    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657fb85584-4wbhr: secret "image-registry-tls" not found
Sep 11 00:58:24.590788 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.590746    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls podName:191b3b23-de69-4658-a4ca-90bdef9cf7e1 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:25.090728064 +0000 UTC m=+34.627379244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls") pod "image-registry-657fb85584-4wbhr" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1") : secret "image-registry-tls" not found
Sep 11 00:58:24.590872 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.590850    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/018f8615-58a3-4dc7-bc26-f5d740868701-config\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.591400 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.591379    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:24.591489 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.591474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27c3cca3-b650-4a88-a6c6-47a747ec800e-config\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.591544 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.591508    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls podName:bde7aa89-7b63-4dfb-b48e-12ea9734ab7d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:25.091488937 +0000 UTC m=+34.628140134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-qjmm2" (UID: "bde7aa89-7b63-4dfb-b48e-12ea9734ab7d") : secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:24.591940 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.591920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.592405 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.592379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/018f8615-58a3-4dc7-bc26-f5d740868701-trusted-ca\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.594145 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.594117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a10e86f3-6bcd-42da-a580-e218495a161b-serving-cert\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.594145 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.594129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/018f8615-58a3-4dc7-bc26-f5d740868701-serving-cert\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.594337 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.594258    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-image-registry-private-configuration\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.594398 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.594354    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27c3cca3-b650-4a88-a6c6-47a747ec800e-serving-cert\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.594398 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.594365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-installation-pull-secrets\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.599054 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.599011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpxv\" (UniqueName: \"kubernetes.io/projected/27c3cca3-b650-4a88-a6c6-47a747ec800e-kube-api-access-rzpxv\") pod \"service-ca-operator-cf7c776b7-wfjl6\" (UID: \"27c3cca3-b650-4a88-a6c6-47a747ec800e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.599162 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.599142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwvw\" (UniqueName: \"kubernetes.io/projected/31932c40-b8a9-4ab3-8901-6d3d970aa7c5-kube-api-access-tfwvw\") pod \"volume-data-source-validator-86695b7cc7-qvzgd\" (UID: \"31932c40-b8a9-4ab3-8901-6d3d970aa7c5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd"
Sep 11 00:58:24.599691 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.599473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9glmv\" (UniqueName: \"kubernetes.io/projected/018f8615-58a3-4dc7-bc26-f5d740868701-kube-api-access-9glmv\") pod \"console-operator-696884fb9c-mrjfb\" (UID: \"018f8615-58a3-4dc7-bc26-f5d740868701\") " pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.601785 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.601762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-bound-sa-token\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.602775 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.602744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4ws\" (UniqueName: \"kubernetes.io/projected/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-kube-api-access-bd4ws\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:24.603718 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.603673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmtk\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-kube-api-access-vbmtk\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:24.604430 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.604407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6qg\" (UniqueName: \"kubernetes.io/projected/a10e86f3-6bcd-42da-a580-e218495a161b-kube-api-access-hh6qg\") pod \"insights-operator-5844b856db-km5j6\" (UID: \"a10e86f3-6bcd-42da-a580-e218495a161b\") " pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.648097 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.648067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-km5j6"
Sep 11 00:58:24.676437 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.676414    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:24.689985 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.689958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd"
Sep 11 00:58:24.690373 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-tmp-dir\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.690373 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9485f75-5961-4809-bb52-ff9d335b4b39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4977b89b-757w4\" (UID: \"e9485f75-5961-4809-bb52-ff9d335b4b39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"
Sep 11 00:58:24.690373 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk656\" (UniqueName: \"kubernetes.io/projected/e9485f75-5961-4809-bb52-ff9d335b4b39-kube-api-access-xk656\") pod \"managed-serviceaccount-addon-agent-7b4977b89b-757w4\" (UID: \"e9485f75-5961-4809-bb52-ff9d335b4b39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"
Sep 11 00:58:24.690373 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-hub\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.690373 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szbc\" (UniqueName: \"kubernetes.io/projected/cefebf34-9896-4f29-84b3-596e6bbb0b9e-kube-api-access-4szbc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.690373 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79c84350-cf67-43ba-9abb-dd3a881108d0-tmp\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hb5x\" (UniqueName: \"kubernetes.io/projected/0429739a-4188-4eb1-b97a-8f7c2e172a71-kube-api-access-4hb5x\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cefebf34-9896-4f29-84b3-596e6bbb0b9e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690490    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-tmp-dir\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.690541    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:25.190520335 +0000 UTC m=+34.727171513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : configmap references non-existent config key: service-ca.crt
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.690580    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.690629    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls podName:583bbbc2-81a6-4a5f-b224-46ec7e1ecd86 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:25.190611117 +0000 UTC m=+34.727262321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls") pod "dns-default-rncnx" (UID: "583bbbc2-81a6-4a5f-b224-46ec7e1ecd86") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:24.690672 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfcx\" (UniqueName: \"kubernetes.io/projected/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-kube-api-access-dhfcx\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.690775    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79c84350-cf67-43ba-9abb-dd3a881108d0-tmp\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.690812    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls podName:e69cf70a-9cc8-4ddc-820f-cca3e67c2a18 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:25.190799433 +0000 UTC m=+34.727450628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bxnz8" (UID: "e69cf70a-9cc8-4ddc-820f-cca3e67c2a18") : secret "samples-operator-tls" not found
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-stats-auth\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zpr\" (UniqueName: \"kubernetes.io/projected/557f06ae-6ab3-4488-9985-4d101c89a35e-kube-api-access-x9zpr\") pod \"network-check-source-6f8b768455-dvh6b\" (UID: \"557f06ae-6ab3-4488-9985-4d101c89a35e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0429739a-4188-4eb1-b97a-8f7c2e172a71-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.690958    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cefebf34-9896-4f29-84b3-596e6bbb0b9e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.690994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2nnm\" (UniqueName: \"kubernetes.io/projected/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-kube-api-access-s2nnm\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:24.691001    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:25.19098785 +0000 UTC m=+34.727639050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : secret "router-metrics-certs-default" not found
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691033    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjcld\" (UniqueName: \"kubernetes.io/projected/d23e8729-271c-4456-a421-4ef108717362-kube-api-access-xjcld\") pod \"ingress-canary-drvkn\" (UID: \"d23e8729-271c-4456-a421-4ef108717362\") " pod="openshift-ingress-canary/ingress-canary-drvkn"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.691154 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cefebf34-9896-4f29-84b3-596e6bbb0b9e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.691880 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-default-certificate\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.691880 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9kg\" (UniqueName: \"kubernetes.io/projected/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-kube-api-access-dg9kg\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:24.691880 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-config-volume\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.691880 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-ca\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.691880 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hljh\" (UniqueName: \"kubernetes.io/projected/79c84350-cf67-43ba-9abb-dd3a881108d0-kube-api-access-6hljh\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.691880 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d23e8729-271c-4456-a421-4ef108717362-cert\") pod \"ingress-canary-drvkn\" (UID: \"d23e8729-271c-4456-a421-4ef108717362\") " pod="openshift-ingress-canary/ingress-canary-drvkn"
Sep 11 00:58:24.691880 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/79c84350-cf67-43ba-9abb-dd3a881108d0-klusterlet-config\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.691880 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.691689    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-config-volume\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.693748 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.693697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cefebf34-9896-4f29-84b3-596e6bbb0b9e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.694024 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.693988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-default-certificate\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.694140 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.694040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-stats-auth\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.694196 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.694171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/79c84350-cf67-43ba-9abb-dd3a881108d0-klusterlet-config\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.704670 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.704641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hljh\" (UniqueName: \"kubernetes.io/projected/79c84350-cf67-43ba-9abb-dd3a881108d0-kube-api-access-6hljh\") pod \"klusterlet-addon-workmgr-d7bbb6866-pcpg5\" (UID: \"79c84350-cf67-43ba-9abb-dd3a881108d0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.705182 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.705161    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9kg\" (UniqueName: \"kubernetes.io/projected/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-kube-api-access-dg9kg\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:24.705574 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.704693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2nnm\" (UniqueName: \"kubernetes.io/projected/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-kube-api-access-s2nnm\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:24.706120 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.705908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfcx\" (UniqueName: \"kubernetes.io/projected/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-kube-api-access-dhfcx\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:24.706120 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.706001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szbc\" (UniqueName: \"kubernetes.io/projected/cefebf34-9896-4f29-84b3-596e6bbb0b9e-kube-api-access-4szbc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-k9cb4\" (UID: \"cefebf34-9896-4f29-84b3-596e6bbb0b9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.710235 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.710215    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"
Sep 11 00:58:24.746576 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.746555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"
Sep 11 00:58:24.792654 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d23e8729-271c-4456-a421-4ef108717362-cert\") pod \"ingress-canary-drvkn\" (UID: \"d23e8729-271c-4456-a421-4ef108717362\") " pod="openshift-ingress-canary/ingress-canary-drvkn"
Sep 11 00:58:24.792748 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9485f75-5961-4809-bb52-ff9d335b4b39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4977b89b-757w4\" (UID: \"e9485f75-5961-4809-bb52-ff9d335b4b39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"
Sep 11 00:58:24.792748 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk656\" (UniqueName: \"kubernetes.io/projected/e9485f75-5961-4809-bb52-ff9d335b4b39-kube-api-access-xk656\") pod \"managed-serviceaccount-addon-agent-7b4977b89b-757w4\" (UID: \"e9485f75-5961-4809-bb52-ff9d335b4b39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"
Sep 11 00:58:24.792748 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-hub\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.792908 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hb5x\" (UniqueName: \"kubernetes.io/projected/0429739a-4188-4eb1-b97a-8f7c2e172a71-kube-api-access-4hb5x\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.792908 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.792908 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zpr\" (UniqueName: \"kubernetes.io/projected/557f06ae-6ab3-4488-9985-4d101c89a35e-kube-api-access-x9zpr\") pod \"network-check-source-6f8b768455-dvh6b\" (UID: \"557f06ae-6ab3-4488-9985-4d101c89a35e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b"
Sep 11 00:58:24.793039 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0429739a-4188-4eb1-b97a-8f7c2e172a71-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.793039 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjcld\" (UniqueName: \"kubernetes.io/projected/d23e8729-271c-4456-a421-4ef108717362-kube-api-access-xjcld\") pod \"ingress-canary-drvkn\" (UID: \"d23e8729-271c-4456-a421-4ef108717362\") " pod="openshift-ingress-canary/ingress-canary-drvkn"
Sep 11 00:58:24.793039 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.793039 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.792977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-ca\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.794061 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.794032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0429739a-4188-4eb1-b97a-8f7c2e172a71-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.795726 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.795697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9485f75-5961-4809-bb52-ff9d335b4b39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4977b89b-757w4\" (UID: \"e9485f75-5961-4809-bb52-ff9d335b4b39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"
Sep 11 00:58:24.795822 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.795702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.795884 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.795856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.795943 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.795921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d23e8729-271c-4456-a421-4ef108717362-cert\") pod \"ingress-canary-drvkn\" (UID: \"d23e8729-271c-4456-a421-4ef108717362\") " pod="openshift-ingress-canary/ingress-canary-drvkn"
Sep 11 00:58:24.796115 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.796090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-hub\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.796255 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.796241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0429739a-4188-4eb1-b97a-8f7c2e172a71-ca\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.800849 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.800793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:24.801604 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.801586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hb5x\" (UniqueName: \"kubernetes.io/projected/0429739a-4188-4eb1-b97a-8f7c2e172a71-kube-api-access-4hb5x\") pod \"cluster-proxy-proxy-agent-cb86697bc-64dwz\" (UID: \"0429739a-4188-4eb1-b97a-8f7c2e172a71\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.801925 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.801899    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zpr\" (UniqueName: \"kubernetes.io/projected/557f06ae-6ab3-4488-9985-4d101c89a35e-kube-api-access-x9zpr\") pod \"network-check-source-6f8b768455-dvh6b\" (UID: \"557f06ae-6ab3-4488-9985-4d101c89a35e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b"
Sep 11 00:58:24.802017 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.801956    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk656\" (UniqueName: \"kubernetes.io/projected/e9485f75-5961-4809-bb52-ff9d335b4b39-kube-api-access-xk656\") pod \"managed-serviceaccount-addon-agent-7b4977b89b-757w4\" (UID: \"e9485f75-5961-4809-bb52-ff9d335b4b39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"
Sep 11 00:58:24.802070 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.802016    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjcld\" (UniqueName: \"kubernetes.io/projected/d23e8729-271c-4456-a421-4ef108717362-kube-api-access-xjcld\") pod \"ingress-canary-drvkn\" (UID: \"d23e8729-271c-4456-a421-4ef108717362\") " pod="openshift-ingress-canary/ingress-canary-drvkn"
Sep 11 00:58:24.854845 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.854812    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"
Sep 11 00:58:24.862002 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.861974    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 00:58:24.884322 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.884289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b"
Sep 11 00:58:24.891462 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:24.891436    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drvkn"
Sep 11 00:58:25.095849 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:25.095814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:25.095849 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:25.095852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:25.096073 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.095956    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:25.096073 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.095963    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:25.096073 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.095981    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657fb85584-4wbhr: secret "image-registry-tls" not found
Sep 11 00:58:25.096073 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.096021    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls podName:bde7aa89-7b63-4dfb-b48e-12ea9734ab7d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.096005204 +0000 UTC m=+35.632656388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-qjmm2" (UID: "bde7aa89-7b63-4dfb-b48e-12ea9734ab7d") : secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:25.096073 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.096037    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls podName:191b3b23-de69-4658-a4ca-90bdef9cf7e1 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.096029906 +0000 UTC m=+35.632681085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls") pod "image-registry-657fb85584-4wbhr" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1") : secret "image-registry-tls" not found
Sep 11 00:58:25.196854 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:25.196821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:25.197036 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:25.196872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:25.197036 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.196969    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:25.197036 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.196981    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:25.197036 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:25.197009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:25.197036 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.197032    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls podName:e69cf70a-9cc8-4ddc-820f-cca3e67c2a18 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.197013598 +0000 UTC m=+35.733664797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bxnz8" (UID: "e69cf70a-9cc8-4ddc-820f-cca3e67c2a18") : secret "samples-operator-tls" not found
Sep 11 00:58:25.197273 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.197079    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 00:58:25.197273 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.197088    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls podName:583bbbc2-81a6-4a5f-b224-46ec7e1ecd86 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.197070777 +0000 UTC m=+35.733721961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls") pod "dns-default-rncnx" (UID: "583bbbc2-81a6-4a5f-b224-46ec7e1ecd86") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:25.197273 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.197121    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.197109434 +0000 UTC m=+35.733760616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : secret "router-metrics-certs-default" not found
Sep 11 00:58:25.197273 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:25.197240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:25.197442 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:25.197374    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.197364987 +0000 UTC m=+35.734016179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : configmap references non-existent config key: service-ca.crt
Sep 11 00:58:26.030735 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.030604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:26.031110 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.030613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:26.031110 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.030624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:26.033359 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.033337    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 00:58:26.033573 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.033554    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 00:58:26.034760 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.034740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zpsrm\""
Sep 11 00:58:26.034760 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.034754    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h86gq\""
Sep 11 00:58:26.070557 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.070534    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"]
Sep 11 00:58:26.074817 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.074792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-km5j6"]
Sep 11 00:58:26.091496 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.091473    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4"]
Sep 11 00:58:26.094052 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.094029    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"]
Sep 11 00:58:26.095121 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.095098    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4"]
Sep 11 00:58:26.103284 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.103264    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mrjfb"]
Sep 11 00:58:26.107455 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.107424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:26.107519 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.107463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:26.107754 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.107582    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:26.107754 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.107596    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:26.107754 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.107647    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657fb85584-4wbhr: secret "image-registry-tls" not found
Sep 11 00:58:26.107754 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.107637    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls podName:bde7aa89-7b63-4dfb-b48e-12ea9734ab7d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:28.107618362 +0000 UTC m=+37.644269559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-qjmm2" (UID: "bde7aa89-7b63-4dfb-b48e-12ea9734ab7d") : secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:26.107754 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.107713    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls podName:191b3b23-de69-4658-a4ca-90bdef9cf7e1 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:28.107698633 +0000 UTC m=+37.644349815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls") pod "image-registry-657fb85584-4wbhr" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1") : secret "image-registry-tls" not found
Sep 11 00:58:26.108095 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.108070    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd"]
Sep 11 00:58:26.109090 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.109068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b"]
Sep 11 00:58:26.110055 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.110035    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drvkn"]
Sep 11 00:58:26.117577 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.117558    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6"]
Sep 11 00:58:26.120602 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:26.120562    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9485f75_5961_4809_bb52_ff9d335b4b39.slice/crio-bacc90eed335ddc2836b0cd21b383b5950364d1d398e92ec0922db11df5877c9 WatchSource:0}: Error finding container bacc90eed335ddc2836b0cd21b383b5950364d1d398e92ec0922db11df5877c9: Status 404 returned error can't find the container with id bacc90eed335ddc2836b0cd21b383b5950364d1d398e92ec0922db11df5877c9
Sep 11 00:58:26.121257 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:26.121179    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c84350_cf67_43ba_9abb_dd3a881108d0.slice/crio-6bb037e8a40670171fd222dbc6de650991fbbfe7290980bcc621ca07c2965959 WatchSource:0}: Error finding container 6bb037e8a40670171fd222dbc6de650991fbbfe7290980bcc621ca07c2965959: Status 404 returned error can't find the container with id 6bb037e8a40670171fd222dbc6de650991fbbfe7290980bcc621ca07c2965959
Sep 11 00:58:26.122506 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:26.122486    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c3cca3_b650_4a88_a6c6_47a747ec800e.slice/crio-8b1bbabe2796ae18f2fe8d7b20677afa2255660d23153f9884fd4391cc3e95ed WatchSource:0}: Error finding container 8b1bbabe2796ae18f2fe8d7b20677afa2255660d23153f9884fd4391cc3e95ed: Status 404 returned error can't find the container with id 8b1bbabe2796ae18f2fe8d7b20677afa2255660d23153f9884fd4391cc3e95ed
Sep 11 00:58:26.123727 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:26.123681    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0429739a_4188_4eb1_b97a_8f7c2e172a71.slice/crio-728931b9764267ff012110d74ca00842c26f2ddc165103c2edb50fe6c723446f WatchSource:0}: Error finding container 728931b9764267ff012110d74ca00842c26f2ddc165103c2edb50fe6c723446f: Status 404 returned error can't find the container with id 728931b9764267ff012110d74ca00842c26f2ddc165103c2edb50fe6c723446f
Sep 11 00:58:26.124868 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:26.124847    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10e86f3_6bcd_42da_a580_e218495a161b.slice/crio-2994192eed7927bd15122bcfd0319a80bbec3ebacb0d8bf6c864d8fd04349078 WatchSource:0}: Error finding container 2994192eed7927bd15122bcfd0319a80bbec3ebacb0d8bf6c864d8fd04349078: Status 404 returned error can't find the container with id 2994192eed7927bd15122bcfd0319a80bbec3ebacb0d8bf6c864d8fd04349078
Sep 11 00:58:26.125743 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:26.125726    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23e8729_271c_4456_a421_4ef108717362.slice/crio-9cd381385f559e4f108fb6068cdc4fc09fddb470eafcf24756066d69beba3c2c WatchSource:0}: Error finding container 9cd381385f559e4f108fb6068cdc4fc09fddb470eafcf24756066d69beba3c2c: Status 404 returned error can't find the container with id 9cd381385f559e4f108fb6068cdc4fc09fddb470eafcf24756066d69beba3c2c
Sep 11 00:58:26.126194 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:26.126173    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefebf34_9896_4f29_84b3_596e6bbb0b9e.slice/crio-fbb0d09f45661076829b57d06a9cc8b37f7a27a19cb5edea59198e369eb8d9d8 WatchSource:0}: Error finding container fbb0d09f45661076829b57d06a9cc8b37f7a27a19cb5edea59198e369eb8d9d8: Status 404 returned error can't find the container with id fbb0d09f45661076829b57d06a9cc8b37f7a27a19cb5edea59198e369eb8d9d8
Sep 11 00:58:26.208934 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.208906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:26.209028 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.208990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:26.209076 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.209027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:26.209076 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.209047    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:28.209028829 +0000 UTC m=+37.745680026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : configmap references non-existent config key: service-ca.crt
Sep 11 00:58:26.209157 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.209080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:26.209157 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.209088    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:26.209157 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.209101    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:26.209157 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.209147    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls podName:583bbbc2-81a6-4a5f-b224-46ec7e1ecd86 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:28.209130217 +0000 UTC m=+37.745781410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls") pod "dns-default-rncnx" (UID: "583bbbc2-81a6-4a5f-b224-46ec7e1ecd86") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:26.209281 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.209165    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls podName:e69cf70a-9cc8-4ddc-820f-cca3e67c2a18 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:28.209156466 +0000 UTC m=+37.745807648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bxnz8" (UID: "e69cf70a-9cc8-4ddc-820f-cca3e67c2a18") : secret "samples-operator-tls" not found
Sep 11 00:58:26.209606 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.209593    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 00:58:26.209647 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:26.209631    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:28.209622457 +0000 UTC m=+37.746273636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : secret "router-metrics-certs-default" not found
Sep 11 00:58:26.332498 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.332472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5" event={"ID":"79c84350-cf67-43ba-9abb-dd3a881108d0","Type":"ContainerStarted","Data":"6bb037e8a40670171fd222dbc6de650991fbbfe7290980bcc621ca07c2965959"}
Sep 11 00:58:26.333672 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.333646    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b" event={"ID":"557f06ae-6ab3-4488-9985-4d101c89a35e","Type":"ContainerStarted","Data":"a2a5e956c5c2c8239c4266379028a9bd6588ac22de2ad75b781ef5f8bf157816"}
Sep 11 00:58:26.338014 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.335101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-km5j6" event={"ID":"a10e86f3-6bcd-42da-a580-e218495a161b","Type":"ContainerStarted","Data":"2994192eed7927bd15122bcfd0319a80bbec3ebacb0d8bf6c864d8fd04349078"}
Sep 11 00:58:26.339212 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.339190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6" event={"ID":"27c3cca3-b650-4a88-a6c6-47a747ec800e","Type":"ContainerStarted","Data":"8b1bbabe2796ae18f2fe8d7b20677afa2255660d23153f9884fd4391cc3e95ed"}
Sep 11 00:58:26.340294 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.340275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drvkn" event={"ID":"d23e8729-271c-4456-a421-4ef108717362","Type":"ContainerStarted","Data":"9cd381385f559e4f108fb6068cdc4fc09fddb470eafcf24756066d69beba3c2c"}
Sep 11 00:58:26.341339 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.341316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4" event={"ID":"e9485f75-5961-4809-bb52-ff9d335b4b39","Type":"ContainerStarted","Data":"bacc90eed335ddc2836b0cd21b383b5950364d1d398e92ec0922db11df5877c9"}
Sep 11 00:58:26.342261 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.342242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4" event={"ID":"cefebf34-9896-4f29-84b3-596e6bbb0b9e","Type":"ContainerStarted","Data":"fbb0d09f45661076829b57d06a9cc8b37f7a27a19cb5edea59198e369eb8d9d8"}
Sep 11 00:58:26.343240 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.343221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" event={"ID":"018f8615-58a3-4dc7-bc26-f5d740868701","Type":"ContainerStarted","Data":"c73317f099e28240452a4f5e60cf980ac0472d0fb373284d8170e15e5c64d132"}
Sep 11 00:58:26.345896 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.345875    2583 generic.go:356] "Generic (PLEG): container finished" podID="68c89915-cc67-481d-bf53-0a2656dd28bc" containerID="02baa05f56ec8e17c1f033ca583c8b975321614cb674d09ebf4392b52794a61c" exitCode=0
Sep 11 00:58:26.345975 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.345953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rb67" event={"ID":"68c89915-cc67-481d-bf53-0a2656dd28bc","Type":"ContainerDied","Data":"02baa05f56ec8e17c1f033ca583c8b975321614cb674d09ebf4392b52794a61c"}
Sep 11 00:58:26.347054 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.347034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd" event={"ID":"31932c40-b8a9-4ab3-8901-6d3d970aa7c5","Type":"ContainerStarted","Data":"e30c30200d717c3c840a7ced0262af048046ac25ee4d1f37102c75de5ea020be"}
Sep 11 00:58:26.348101 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:26.348083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" event={"ID":"0429739a-4188-4eb1-b97a-8f7c2e172a71","Type":"ContainerStarted","Data":"728931b9764267ff012110d74ca00842c26f2ddc165103c2edb50fe6c723446f"}
Sep 11 00:58:27.386442 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:27.386170    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqpsr_e47f1012-5a54-441e-8fe1-f12d68b88038/konnectivity-agent/0.log" limit=1024
Sep 11 00:58:27.407509 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:27.407387    2583 generic.go:356] "Generic (PLEG): container finished" podID="68c89915-cc67-481d-bf53-0a2656dd28bc" containerID="ba40a5bd73324810bedb344eca2f13efdd513e59a6224468363d0ca2b45ae24e" exitCode=0
Sep 11 00:58:27.407509 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:27.407452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rb67" event={"ID":"68c89915-cc67-481d-bf53-0a2656dd28bc","Type":"ContainerDied","Data":"ba40a5bd73324810bedb344eca2f13efdd513e59a6224468363d0ca2b45ae24e"}
Sep 11 00:58:28.133072 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.132197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:28.133072 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.132252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:28.133072 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.132454    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:28.133072 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.132521    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls podName:bde7aa89-7b63-4dfb-b48e-12ea9734ab7d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:32.132500803 +0000 UTC m=+41.669151989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-qjmm2" (UID: "bde7aa89-7b63-4dfb-b48e-12ea9734ab7d") : secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:28.133072 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.132965    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:28.133072 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.132984    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657fb85584-4wbhr: secret "image-registry-tls" not found
Sep 11 00:58:28.133072 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.133033    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls podName:191b3b23-de69-4658-a4ca-90bdef9cf7e1 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:32.133016759 +0000 UTC m=+41.669667962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls") pod "image-registry-657fb85584-4wbhr" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1") : secret "image-registry-tls" not found
Sep 11 00:58:28.233980 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.233947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:28.234152 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.234027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:28.234152 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.234062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:28.234152 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.234093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:28.234324 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.234282    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 00:58:28.234382 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.234360    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:32.234339155 +0000 UTC m=+41.770990355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : secret "router-metrics-certs-default" not found
Sep 11 00:58:28.236232 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.234766    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:32.234751523 +0000 UTC m=+41.771402707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : configmap references non-existent config key: service-ca.crt
Sep 11 00:58:28.236232 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.234843    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:28.236232 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.234895    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls podName:583bbbc2-81a6-4a5f-b224-46ec7e1ecd86 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:32.234866654 +0000 UTC m=+41.771517847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls") pod "dns-default-rncnx" (UID: "583bbbc2-81a6-4a5f-b224-46ec7e1ecd86") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:28.236232 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.234946    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:28.236232 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:28.234977    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls podName:e69cf70a-9cc8-4ddc-820f-cca3e67c2a18 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:32.234966519 +0000 UTC m=+41.771617713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bxnz8" (UID: "e69cf70a-9cc8-4ddc-820f-cca3e67c2a18") : secret "samples-operator-tls" not found
Sep 11 00:58:28.468142 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.468058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rb67" event={"ID":"68c89915-cc67-481d-bf53-0a2656dd28bc","Type":"ContainerStarted","Data":"29bff16f2c5a6f1a31d81f1247161520b168097a008dcf303d2846b34e2daefd"}
Sep 11 00:58:28.955485 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.955188    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6rb67" podStartSLOduration=7.507520255 podStartE2EDuration="37.955162974s" totalImagesPullingTime="30.447642719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:28.493091658 +0000 UTC m=+38.029742857" watchObservedRunningTime="2026-09-11 00:58:28.955162974 +0000 UTC m=+38.491814177"
Sep 11 00:58:28.956104 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.956081    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"]
Sep 11 00:58:28.959171 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.958485    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:28.961789 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.961280    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 00:58:28.961789 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.961538    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 00:58:28.962048 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.962013    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hbvts\""
Sep 11 00:58:28.972851 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:28.972828    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"]
Sep 11 00:58:29.054168 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:29.054126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:29.054364 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:29.054260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/614f95d1-510e-4863-ac36-e2aa7a01ccef-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:29.154873 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:29.154830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:29.155099 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:29.155054    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/614f95d1-510e-4863-ac36-e2aa7a01ccef-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:29.155256 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:29.155237    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 00:58:29.155862 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:29.155675    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert podName:614f95d1-510e-4863-ac36-e2aa7a01ccef nodeName:}" failed. No retries permitted until 2026-09-11 00:58:29.655650577 +0000 UTC m=+39.192301771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c2z44" (UID: "614f95d1-510e-4863-ac36-e2aa7a01ccef") : secret "networking-console-plugin-cert" not found
Sep 11 00:58:29.156074 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:29.156049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/614f95d1-510e-4863-ac36-e2aa7a01ccef-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:29.661689 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:29.661647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:29.662144 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:29.662035    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 00:58:29.662144 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:29.662124    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert podName:614f95d1-510e-4863-ac36-e2aa7a01ccef nodeName:}" failed. No retries permitted until 2026-09-11 00:58:30.662103184 +0000 UTC m=+40.198754574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c2z44" (UID: "614f95d1-510e-4863-ac36-e2aa7a01ccef") : secret "networking-console-plugin-cert" not found
Sep 11 00:58:30.673923 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:30.673880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:30.674410 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:30.674054    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 00:58:30.674410 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:30.674133    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert podName:614f95d1-510e-4863-ac36-e2aa7a01ccef nodeName:}" failed. No retries permitted until 2026-09-11 00:58:32.674117395 +0000 UTC m=+42.210768578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c2z44" (UID: "614f95d1-510e-4863-ac36-e2aa7a01ccef") : secret "networking-console-plugin-cert" not found
Sep 11 00:58:32.189889 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:32.189856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:32.189889 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:32.189893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:32.190359 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.189998    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:32.190359 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.190046    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls podName:bde7aa89-7b63-4dfb-b48e-12ea9734ab7d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:40.190033206 +0000 UTC m=+49.726684407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-qjmm2" (UID: "bde7aa89-7b63-4dfb-b48e-12ea9734ab7d") : secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:32.190359 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.190088    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:32.190359 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.190107    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657fb85584-4wbhr: secret "image-registry-tls" not found
Sep 11 00:58:32.190359 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.190154    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls podName:191b3b23-de69-4658-a4ca-90bdef9cf7e1 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:40.190142831 +0000 UTC m=+49.726794032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls") pod "image-registry-657fb85584-4wbhr" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1") : secret "image-registry-tls" not found
Sep 11 00:58:32.290565 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:32.290530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:32.290565 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:32.290568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:32.290730 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.290656    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 00:58:32.290730 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.290674    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:32.290730 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.290701    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:40.290686358 +0000 UTC m=+49.827337536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : secret "router-metrics-certs-default" not found
Sep 11 00:58:32.290730 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.290722    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls podName:e69cf70a-9cc8-4ddc-820f-cca3e67c2a18 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:40.290711551 +0000 UTC m=+49.827362744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bxnz8" (UID: "e69cf70a-9cc8-4ddc-820f-cca3e67c2a18") : secret "samples-operator-tls" not found
Sep 11 00:58:32.290856 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:32.290801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:32.290856 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:32.290840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:32.290927 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.290916    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:32.290960 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.290950    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:40.290934396 +0000 UTC m=+49.827585590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : configmap references non-existent config key: service-ca.crt
Sep 11 00:58:32.291000 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.290973    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls podName:583bbbc2-81a6-4a5f-b224-46ec7e1ecd86 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:40.290963732 +0000 UTC m=+49.827614911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls") pod "dns-default-rncnx" (UID: "583bbbc2-81a6-4a5f-b224-46ec7e1ecd86") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:32.694700 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:32.694658    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:32.694878 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.694811    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 00:58:32.694939 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:32.694877    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert podName:614f95d1-510e-4863-ac36-e2aa7a01ccef nodeName:}" failed. No retries permitted until 2026-09-11 00:58:36.694864249 +0000 UTC m=+46.231515434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c2z44" (UID: "614f95d1-510e-4863-ac36-e2aa7a01ccef") : secret "networking-console-plugin-cert" not found
Sep 11 00:58:36.733097 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:36.733059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:36.733588 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:36.733262    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 00:58:36.733588 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:36.733369    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert podName:614f95d1-510e-4863-ac36-e2aa7a01ccef nodeName:}" failed. No retries permitted until 2026-09-11 00:58:44.733346731 +0000 UTC m=+54.269997941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c2z44" (UID: "614f95d1-510e-4863-ac36-e2aa7a01ccef") : secret "networking-console-plugin-cert" not found
Sep 11 00:58:40.267481 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:40.267432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:40.267481 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:40.267489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:40.267990 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.267593    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:40.267990 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.267669    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls podName:bde7aa89-7b63-4dfb-b48e-12ea9734ab7d nodeName:}" failed. No retries permitted until 2026-09-11 00:58:56.267652906 +0000 UTC m=+65.804304098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-qjmm2" (UID: "bde7aa89-7b63-4dfb-b48e-12ea9734ab7d") : secret "cluster-monitoring-operator-tls" not found
Sep 11 00:58:40.267990 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.267596    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 00:58:40.267990 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.267701    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657fb85584-4wbhr: secret "image-registry-tls" not found
Sep 11 00:58:40.267990 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.267763    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls podName:191b3b23-de69-4658-a4ca-90bdef9cf7e1 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:56.267748286 +0000 UTC m=+65.804399487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls") pod "image-registry-657fb85584-4wbhr" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1") : secret "image-registry-tls" not found
Sep 11 00:58:40.368396 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:40.368367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:40.368474 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:40.368421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:40.368508 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:40.368467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:40.368544 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.368507    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 00:58:40.368578 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.368555    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls podName:583bbbc2-81a6-4a5f-b224-46ec7e1ecd86 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:56.368543195 +0000 UTC m=+65.905194374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls") pod "dns-default-rncnx" (UID: "583bbbc2-81a6-4a5f-b224-46ec7e1ecd86") : secret "dns-default-metrics-tls" not found
Sep 11 00:58:40.368621 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:40.368508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:40.368621 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.368584    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 00:58:40.368691 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.368635    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:56.368623715 +0000 UTC m=+65.905274897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : secret "router-metrics-certs-default" not found
Sep 11 00:58:40.368733 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.368688    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 00:58:40.368769 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.368695    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle podName:dfdd4c42-aa61-4c97-92fd-7cdca51780bd nodeName:}" failed. No retries permitted until 2026-09-11 00:58:56.368683358 +0000 UTC m=+65.905334541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle") pod "router-default-6994df4f88-cwhzx" (UID: "dfdd4c42-aa61-4c97-92fd-7cdca51780bd") : configmap references non-existent config key: service-ca.crt
Sep 11 00:58:40.368769 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.368764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls podName:e69cf70a-9cc8-4ddc-820f-cca3e67c2a18 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:56.368747401 +0000 UTC m=+65.905398584 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-bxnz8" (UID: "e69cf70a-9cc8-4ddc-820f-cca3e67c2a18") : secret "samples-operator-tls" not found
Sep 11 00:58:40.986849 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:40.986823    2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018f8615_58a3_4dc7_bc26_f5d740868701.slice/crio-d63a50fdfd09440064cfdd198a0501c721f91214391caaeb9862703695947389.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 00:58:41.513649 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.513570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b" event={"ID":"557f06ae-6ab3-4488-9985-4d101c89a35e","Type":"ContainerStarted","Data":"2d74b2913f4db6191dcf056fafc4defb6b6e9255dcac005c1cccdca2dc11fb96"}
Sep 11 00:58:41.515647 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.515588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-km5j6" event={"ID":"a10e86f3-6bcd-42da-a580-e218495a161b","Type":"ContainerStarted","Data":"83f5c499226bbf7e3f7717d71deab0bf2353789fdf8931efa42011bb46755776"}
Sep 11 00:58:41.518241 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.517839    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6" event={"ID":"27c3cca3-b650-4a88-a6c6-47a747ec800e","Type":"ContainerStarted","Data":"0ccfeb01664f596a578d3d52d7edb0d7b9f868057a0c742ddf122e0da0559202"}
Sep 11 00:58:41.519692 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.519666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drvkn" event={"ID":"d23e8729-271c-4456-a421-4ef108717362","Type":"ContainerStarted","Data":"d00421581ee3f9accd91e6b408a6c9851fe9acb0d33b92f850319ecc5a0863d5"}
Sep 11 00:58:41.521827 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.521803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4" event={"ID":"e9485f75-5961-4809-bb52-ff9d335b4b39","Type":"ContainerStarted","Data":"24d800b7d740f8e05b91b231714677a51532d107954a00e6c1d5384b633b915d"}
Sep 11 00:58:41.523828 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.523808    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4" event={"ID":"cefebf34-9896-4f29-84b3-596e6bbb0b9e","Type":"ContainerStarted","Data":"c09671f895c5f27cfc86408b902bb714b2380178de3a2d1621063a703d0aec91"}
Sep 11 00:58:41.525772 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.525753    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/0.log"
Sep 11 00:58:41.525898 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.525792    2583 generic.go:356] "Generic (PLEG): container finished" podID="018f8615-58a3-4dc7-bc26-f5d740868701" containerID="d63a50fdfd09440064cfdd198a0501c721f91214391caaeb9862703695947389" exitCode=255
Sep 11 00:58:41.525898 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.525872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" event={"ID":"018f8615-58a3-4dc7-bc26-f5d740868701","Type":"ContainerDied","Data":"d63a50fdfd09440064cfdd198a0501c721f91214391caaeb9862703695947389"}
Sep 11 00:58:41.526085 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.526068    2583 scope.go:117] "RemoveContainer" containerID="d63a50fdfd09440064cfdd198a0501c721f91214391caaeb9862703695947389"
Sep 11 00:58:41.529013 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.528938    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd" event={"ID":"31932c40-b8a9-4ab3-8901-6d3d970aa7c5","Type":"ContainerStarted","Data":"2bff0803dc2cb645425deecb6e2ecec6c08870d4d4ad11a8a988f5db527e8dda"}
Sep 11 00:58:41.529676 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.529633    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dvh6b" podStartSLOduration=25.867809055 podStartE2EDuration="40.529617156s" totalImagesPullingTime="14.661808101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.528212707 +0000 UTC m=+51.064863909" watchObservedRunningTime="2026-09-11 00:58:41.529617156 +0000 UTC m=+51.066268352"
Sep 11 00:58:41.531485 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.531466    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" event={"ID":"0429739a-4188-4eb1-b97a-8f7c2e172a71","Type":"ContainerStarted","Data":"2c945c1d5581241ae1b543976f12ae0b7943472327ac0ea379b39fc2121335a9"}
Sep 11 00:58:41.531638 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.531620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" event={"ID":"0429739a-4188-4eb1-b97a-8f7c2e172a71","Type":"ContainerStarted","Data":"2debdeb8332f64fbcf4b83214a8689ba97735b2d59182c459f9603e23369de19"}
Sep 11 00:58:41.531757 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.531743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" event={"ID":"0429739a-4188-4eb1-b97a-8f7c2e172a71","Type":"ContainerStarted","Data":"44d6acbe02c55d499d346d79963b791a66824b789ba140d61c559b8b43bb00c4"}
Sep 11 00:58:41.533138 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.533118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5" event={"ID":"79c84350-cf67-43ba-9abb-dd3a881108d0","Type":"ContainerStarted","Data":"fa2a7f451b540dc5896e47b37f5e8aeb4f4aa578320767d80b2f4699383c73e4"}
Sep 11 00:58:41.533363 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.533347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:41.534780 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.534746    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5"
Sep 11 00:58:41.552945 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.543339    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6" podStartSLOduration=24.94875881 podStartE2EDuration="39.543298423s" totalImagesPullingTime="14.594539613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.543186646 +0000 UTC m=+51.079837849" watchObservedRunningTime="2026-09-11 00:58:41.543298423 +0000 UTC m=+51.079949626"
Sep 11 00:58:41.565474 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.565421    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-km5j6" podStartSLOduration=29.973344982 podStartE2EDuration="44.565402805s" totalImagesPullingTime="14.592057823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.564896125 +0000 UTC m=+51.101547325" watchObservedRunningTime="2026-09-11 00:58:41.565402805 +0000 UTC m=+51.102054008"
Sep 11 00:58:41.603496 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.603419    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4" podStartSLOduration=28.01353375 podStartE2EDuration="42.603404905s" totalImagesPullingTime="14.589871155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.602544756 +0000 UTC m=+51.139195958" watchObservedRunningTime="2026-09-11 00:58:41.603404905 +0000 UTC m=+51.140056106"
Sep 11 00:58:41.652874 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.652813    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-drvkn" podStartSLOduration=2.98547807 podStartE2EDuration="17.652794602s" totalImagesPullingTime="14.667316532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.624009439 +0000 UTC m=+51.160660639" watchObservedRunningTime="2026-09-11 00:58:41.652794602 +0000 UTC m=+51.189445803"
Sep 11 00:58:41.653823 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.653121    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4977b89b-757w4" podStartSLOduration=28.978575049 podStartE2EDuration="43.653111483s" totalImagesPullingTime="14.674536434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.652098824 +0000 UTC m=+51.188750031" watchObservedRunningTime="2026-09-11 00:58:41.653111483 +0000 UTC m=+51.189762685"
Sep 11 00:58:41.679292 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.677973    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d7bbb6866-pcpg5" podStartSLOduration=29.004549192 podStartE2EDuration="43.677952346s" totalImagesPullingTime="14.673403154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.677202487 +0000 UTC m=+51.213853690" watchObservedRunningTime="2026-09-11 00:58:41.677952346 +0000 UTC m=+51.214603542"
Sep 11 00:58:41.695127 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.695066    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-qvzgd" podStartSLOduration=35.159356661 podStartE2EDuration="44.695052951s" totalImagesPullingTime="9.53569629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.69382781 +0000 UTC m=+51.230479011" watchObservedRunningTime="2026-09-11 00:58:41.695052951 +0000 UTC m=+51.231704154"
Sep 11 00:58:41.715537 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:41.715474    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" podStartSLOduration=29.02938378 podStartE2EDuration="43.71545493s" totalImagesPullingTime="14.68607115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:41.713704919 +0000 UTC m=+51.250356119" watchObservedRunningTime="2026-09-11 00:58:41.71545493 +0000 UTC m=+51.252106133"
Sep 11 00:58:42.538443 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.538413    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 00:58:42.538884 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.538857    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/0.log"
Sep 11 00:58:42.538949 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.538900    2583 generic.go:356] "Generic (PLEG): container finished" podID="018f8615-58a3-4dc7-bc26-f5d740868701" containerID="8d741af551db1ef1de8bf5134574fe1816b8f9e1b6fb13cfe41eed7fa4c6f737" exitCode=255
Sep 11 00:58:42.539219 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.539192    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" event={"ID":"018f8615-58a3-4dc7-bc26-f5d740868701","Type":"ContainerDied","Data":"8d741af551db1ef1de8bf5134574fe1816b8f9e1b6fb13cfe41eed7fa4c6f737"}
Sep 11 00:58:42.539275 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.539235    2583 scope.go:117] "RemoveContainer" containerID="8d741af551db1ef1de8bf5134574fe1816b8f9e1b6fb13cfe41eed7fa4c6f737"
Sep 11 00:58:42.539463 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:42.539440    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mrjfb_openshift-console-operator(018f8615-58a3-4dc7-bc26-f5d740868701)\"" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" podUID="018f8615-58a3-4dc7-bc26-f5d740868701"
Sep 11 00:58:42.539531 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.539517    2583 scope.go:117] "RemoveContainer" containerID="d63a50fdfd09440064cfdd198a0501c721f91214391caaeb9862703695947389"
Sep 11 00:58:42.782269 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.782228    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tvmm6"]
Sep 11 00:58:42.785317 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.785292    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.787590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.787564    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 00:58:42.787590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.787582    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtm2j\""
Sep 11 00:58:42.787590 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.787572    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 00:58:42.794946 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.794925    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tvmm6"]
Sep 11 00:58:42.895547 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.895510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.895711 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.895564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c50397a-edd8-48e0-87ba-71e8f359f890-crio-socket\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.895711 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.895610    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c50397a-edd8-48e0-87ba-71e8f359f890-data-volume\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.895711 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.895695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4xh\" (UniqueName: \"kubernetes.io/projected/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-api-access-kf4xh\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.895874 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.895811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.997153 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.997118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.997335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.997162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c50397a-edd8-48e0-87ba-71e8f359f890-crio-socket\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.997335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.997210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c50397a-edd8-48e0-87ba-71e8f359f890-data-volume\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.997335 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:42.997242    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:42.997335 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.997255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4xh\" (UniqueName: \"kubernetes.io/projected/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-api-access-kf4xh\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.997335 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:42.997330    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:43.497292645 +0000 UTC m=+53.033943824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:42.997599 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.997385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c50397a-edd8-48e0-87ba-71e8f359f890-crio-socket\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.997599 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.997470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.997599 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:42.997583    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:42.997740 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:42.997607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c50397a-edd8-48e0-87ba-71e8f359f890-data-volume\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:42.997740 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:42.997622    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:43.4976131 +0000 UTC m=+53.034264278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:43.006625 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:43.006601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4xh\" (UniqueName: \"kubernetes.io/projected/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-api-access-kf4xh\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:43.501972 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:43.501940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:43.502132 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:43.502009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:43.502181 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:43.502136    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:43.502214 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:43.502184    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:43.502244 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:43.502223    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:44.50220695 +0000 UTC m=+54.038858142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:43.502280 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:43.502257    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:44.502234157 +0000 UTC m=+54.038885356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:43.548288 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:43.548253    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 00:58:43.548772 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:43.548685    2583 scope.go:117] "RemoveContainer" containerID="8d741af551db1ef1de8bf5134574fe1816b8f9e1b6fb13cfe41eed7fa4c6f737"
Sep 11 00:58:43.548920 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:43.548895    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mrjfb_openshift-console-operator(018f8615-58a3-4dc7-bc26-f5d740868701)\"" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" podUID="018f8615-58a3-4dc7-bc26-f5d740868701"
Sep 11 00:58:44.511681 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:44.511646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:44.511859 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:44.511723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:44.511859 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:44.511803    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:44.511859 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:44.511851    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:44.512009 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:44.511885    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:46.511867852 +0000 UTC m=+56.048519032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:44.512009 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:44.511911    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:46.511898194 +0000 UTC m=+56.048549392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:44.530279 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:44.530260    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7zc7_c41dccbf-cc64-4cbf-8d90-c270f6cd6f95/dns-node-resolver/0.log"
Sep 11 00:58:44.676857 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:44.676825    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:44.677286 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:44.676898    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:44.677392 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:44.677302    2583 scope.go:117] "RemoveContainer" containerID="8d741af551db1ef1de8bf5134574fe1816b8f9e1b6fb13cfe41eed7fa4c6f737"
Sep 11 00:58:44.677541 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:44.677518    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mrjfb_openshift-console-operator(018f8615-58a3-4dc7-bc26-f5d740868701)\"" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" podUID="018f8615-58a3-4dc7-bc26-f5d740868701"
Sep 11 00:58:44.815321 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:44.815280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:58:44.815482 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:44.815435    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 00:58:44.815538 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:44.815506    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert podName:614f95d1-510e-4863-ac36-e2aa7a01ccef nodeName:}" failed. No retries permitted until 2026-09-11 00:59:00.815487712 +0000 UTC m=+70.352138906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c2z44" (UID: "614f95d1-510e-4863-ac36-e2aa7a01ccef") : secret "networking-console-plugin-cert" not found
Sep 11 00:58:45.531438 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:45.531412    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-47kkt_9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35/node-ca/0.log"
Sep 11 00:58:46.529571 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:46.529529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:46.529916 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:46.529612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:46.529916 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:46.529647    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:46.529916 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:46.529688    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drvkn_d23e8729-271c-4456-a421-4ef108717362/serve-healthcheck-canary/0.log"
Sep 11 00:58:46.529916 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:46.529722    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:50.529706282 +0000 UTC m=+60.066357484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:46.529916 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:46.529747    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:46.529916 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:46.529795    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:50.529782618 +0000 UTC m=+60.066433816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:50.326643 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:50.326607    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwgdk"
Sep 11 00:58:50.565899 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:50.565851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:50.566077 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:50.565941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:50.566077 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:50.566008    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:50.566077 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:50.566048    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 00:58:50.566174 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:50.566079    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:58.566062622 +0000 UTC m=+68.102713817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:50.566174 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:50.566108    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:58.566094806 +0000 UTC m=+68.102745991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : configmap "kube-rbac-proxy" not found
Sep 11 00:58:55.712770 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:55.712730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:58:55.713353 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:55.712817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:55.715360 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:55.715340    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 00:58:55.715474 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:55.715341    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 00:58:55.723239 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:55.723219    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 00:58:55.723335 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:55.723303    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs podName:2fca14ea-d910-47a0-9563-3ea2ff660cb6 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:59.723281865 +0000 UTC m=+129.259933058 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs") pod "network-metrics-daemon-qrwkr" (UID: "2fca14ea-d910-47a0-9563-3ea2ff660cb6") : secret "metrics-daemon-secret" not found
Sep 11 00:58:55.725385 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:55.725369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9227d14-ae58-49a3-9795-626c24256f78-original-pull-secret\") pod \"global-pull-secret-syncer-m4sbv\" (UID: \"a9227d14-ae58-49a3-9795-626c24256f78\") " pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:55.741453 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:55.741420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m4sbv"
Sep 11 00:58:55.814031 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:55.814000    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:55.816396 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:55.816374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfb9\" (UniqueName: \"kubernetes.io/projected/ac1b18bb-5604-4428-8d7f-77edd1fb1806-kube-api-access-6jfb9\") pod \"network-check-target-psmdm\" (UID: \"ac1b18bb-5604-4428-8d7f-77edd1fb1806\") " pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:56.011268 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.011201    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m4sbv"]
Sep 11 00:58:56.014232 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:56.014203    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9227d14_ae58_49a3_9795_626c24256f78.slice/crio-c1e2a57b67324743527a0578973457039927ff45dcf3c8058840bef50f863e9e WatchSource:0}: Error finding container c1e2a57b67324743527a0578973457039927ff45dcf3c8058840bef50f863e9e: Status 404 returned error can't find the container with id c1e2a57b67324743527a0578973457039927ff45dcf3c8058840bef50f863e9e
Sep 11 00:58:56.031435 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.031414    2583 scope.go:117] "RemoveContainer" containerID="8d741af551db1ef1de8bf5134574fe1816b8f9e1b6fb13cfe41eed7fa4c6f737"
Sep 11 00:58:56.054767 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.054747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h86gq\""
Sep 11 00:58:56.062688 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.062672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:56.317606 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.317575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:56.317759 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.317628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:56.319988 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.319959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bde7aa89-7b63-4dfb-b48e-12ea9734ab7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-qjmm2\" (UID: \"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:56.320104 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.320088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"image-registry-657fb85584-4wbhr\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") " pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:56.330597 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.330481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-psmdm"]
Sep 11 00:58:56.333579 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:56.333560    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1b18bb_5604_4428_8d7f_77edd1fb1806.slice/crio-9545ba7329bb8dd6036a050fa2ef8020758bb90061a6084a6ee7974ad4def7fd WatchSource:0}: Error finding container 9545ba7329bb8dd6036a050fa2ef8020758bb90061a6084a6ee7974ad4def7fd: Status 404 returned error can't find the container with id 9545ba7329bb8dd6036a050fa2ef8020758bb90061a6084a6ee7974ad4def7fd
Sep 11 00:58:56.418685 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.418663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:56.418857 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.418844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:56.418894 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.418871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:56.418938 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.418891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:56.419595 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.419566    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-service-ca-bundle\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:56.420918 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.420892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfdd4c42-aa61-4c97-92fd-7cdca51780bd-metrics-certs\") pod \"router-default-6994df4f88-cwhzx\" (UID: \"dfdd4c42-aa61-4c97-92fd-7cdca51780bd\") " pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:56.421076 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.421054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/583bbbc2-81a6-4a5f-b224-46ec7e1ecd86-metrics-tls\") pod \"dns-default-rncnx\" (UID: \"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86\") " pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:56.421167 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.421152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e69cf70a-9cc8-4ddc-820f-cca3e67c2a18-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-bxnz8\" (UID: \"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:56.459351 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.459326    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hstjm\""
Sep 11 00:58:56.468091 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.468073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:56.527391 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.527335    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bpb4m\""
Sep 11 00:58:56.535068 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.535048    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"
Sep 11 00:58:56.574454 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.574392    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xsh95\""
Sep 11 00:58:56.583462 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.583433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:56.589924 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.589899    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5gzc2\""
Sep 11 00:58:56.597780 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.597728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"
Sep 11 00:58:56.609681 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.599467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m4sbv" event={"ID":"a9227d14-ae58-49a3-9795-626c24256f78","Type":"ContainerStarted","Data":"c1e2a57b67324743527a0578973457039927ff45dcf3c8058840bef50f863e9e"}
Sep 11 00:58:56.609681 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.602381    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 00:58:56.609681 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.602714    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" event={"ID":"018f8615-58a3-4dc7-bc26-f5d740868701","Type":"ContainerStarted","Data":"d5d13d017e32c5bfc994ef3a5e0626113282f86fcdf19f9c0a6fee0684e05b15"}
Sep 11 00:58:56.609681 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.603681    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:56.611270 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.611245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-psmdm" event={"ID":"ac1b18bb-5604-4428-8d7f-77edd1fb1806","Type":"ContainerStarted","Data":"90dbb2f74bed24aa82330c62e5eb0918e517e44b5d71910ec40fe50774fad896"}
Sep 11 00:58:56.611392 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.611289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-psmdm" event={"ID":"ac1b18bb-5604-4428-8d7f-77edd1fb1806","Type":"ContainerStarted","Data":"9545ba7329bb8dd6036a050fa2ef8020758bb90061a6084a6ee7974ad4def7fd"}
Sep 11 00:58:56.611691 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.611652    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:58:56.625161 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.625106    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" podStartSLOduration=45.03597049 podStartE2EDuration="59.62508787s" totalImagesPullingTime="14.58911738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:56.622749945 +0000 UTC m=+66.159401148" watchObservedRunningTime="2026-09-11 00:58:56.62508787 +0000 UTC m=+66.161739071"
Sep 11 00:58:56.626560 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.626532    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xx7m2\""
Sep 11 00:58:56.636358 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.635861    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rncnx"
Sep 11 00:58:56.639193 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.639081    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-psmdm" podStartSLOduration=65.639062236 podStartE2EDuration="1m5.639062236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:56.638365995 +0000 UTC m=+66.175017197" watchObservedRunningTime="2026-09-11 00:58:56.639062236 +0000 UTC m=+66.175713435"
Sep 11 00:58:56.748031 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.748001    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657fb85584-4wbhr"]
Sep 11 00:58:56.752290 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:56.752254    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191b3b23_de69_4658_a4ca_90bdef9cf7e1.slice/crio-1c92adf51e62e53ba5b6bf37408bc3dd40092a9d3e12430681cb8452d7861f71 WatchSource:0}: Error finding container 1c92adf51e62e53ba5b6bf37408bc3dd40092a9d3e12430681cb8452d7861f71: Status 404 returned error can't find the container with id 1c92adf51e62e53ba5b6bf37408bc3dd40092a9d3e12430681cb8452d7861f71
Sep 11 00:58:56.820368 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.820342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2"]
Sep 11 00:58:56.823176 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:56.823151    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde7aa89_7b63_4dfb_b48e_12ea9734ab7d.slice/crio-60f7f7d10edb5aa44feff2f8da979a638b480559d509ca63bf0c0c3010e37cbe WatchSource:0}: Error finding container 60f7f7d10edb5aa44feff2f8da979a638b480559d509ca63bf0c0c3010e37cbe: Status 404 returned error can't find the container with id 60f7f7d10edb5aa44feff2f8da979a638b480559d509ca63bf0c0c3010e37cbe
Sep 11 00:58:56.873901 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.873833    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6994df4f88-cwhzx"]
Sep 11 00:58:56.876739 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:56.876710    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfdd4c42_aa61_4c97_92fd_7cdca51780bd.slice/crio-7fed127a65283244bf202c8feb479f3c025f8daf462ba277236576ea8c75f7aa WatchSource:0}: Error finding container 7fed127a65283244bf202c8feb479f3c025f8daf462ba277236576ea8c75f7aa: Status 404 returned error can't find the container with id 7fed127a65283244bf202c8feb479f3c025f8daf462ba277236576ea8c75f7aa
Sep 11 00:58:56.894208 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.894042    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8"]
Sep 11 00:58:56.931427 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:56.931352    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rncnx"]
Sep 11 00:58:56.936079 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:58:56.936049    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583bbbc2_81a6_4a5f_b224_46ec7e1ecd86.slice/crio-46613f60c2b7d9c6e16129beeae3f391250b30598d0e9090d9abd9087ea80096 WatchSource:0}: Error finding container 46613f60c2b7d9c6e16129beeae3f391250b30598d0e9090d9abd9087ea80096: Status 404 returned error can't find the container with id 46613f60c2b7d9c6e16129beeae3f391250b30598d0e9090d9abd9087ea80096
Sep 11 00:58:57.604567 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.604166    2583 patch_prober.go:28] interesting pod/console-operator-696884fb9c-mrjfb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.9:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 00:58:57.604567 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.604237    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb" podUID="018f8615-58a3-4dc7-bc26-f5d740868701" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 00:58:57.619288 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.619203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8" event={"ID":"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18","Type":"ContainerStarted","Data":"dd547270b060e433f246725d7b78e083b09abd5719970cd0c30584ed47e92f79"}
Sep 11 00:58:57.623280 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.622266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657fb85584-4wbhr" event={"ID":"191b3b23-de69-4658-a4ca-90bdef9cf7e1","Type":"ContainerStarted","Data":"af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274"}
Sep 11 00:58:57.623280 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.622326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657fb85584-4wbhr" event={"ID":"191b3b23-de69-4658-a4ca-90bdef9cf7e1","Type":"ContainerStarted","Data":"1c92adf51e62e53ba5b6bf37408bc3dd40092a9d3e12430681cb8452d7861f71"}
Sep 11 00:58:57.623280 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.623186    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:58:57.624688 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.624661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rncnx" event={"ID":"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86","Type":"ContainerStarted","Data":"46613f60c2b7d9c6e16129beeae3f391250b30598d0e9090d9abd9087ea80096"}
Sep 11 00:58:57.630410 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.630381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6994df4f88-cwhzx" event={"ID":"dfdd4c42-aa61-4c97-92fd-7cdca51780bd","Type":"ContainerStarted","Data":"f7d045af3b798fb0d501e03c4664d85cfc3cb5e4ab3660788df3e911803c1fb7"}
Sep 11 00:58:57.630527 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.630424    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6994df4f88-cwhzx" event={"ID":"dfdd4c42-aa61-4c97-92fd-7cdca51780bd","Type":"ContainerStarted","Data":"7fed127a65283244bf202c8feb479f3c025f8daf462ba277236576ea8c75f7aa"}
Sep 11 00:58:57.633059 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.633006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2" event={"ID":"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d","Type":"ContainerStarted","Data":"60f7f7d10edb5aa44feff2f8da979a638b480559d509ca63bf0c0c3010e37cbe"}
Sep 11 00:58:57.649149 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.648330    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-657fb85584-4wbhr" podStartSLOduration=66.648288939 podStartE2EDuration="1m6.648288939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:57.64723067 +0000 UTC m=+67.183881872" watchObservedRunningTime="2026-09-11 00:58:57.648288939 +0000 UTC m=+67.184940141"
Sep 11 00:58:57.669236 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.668831    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6994df4f88-cwhzx" podStartSLOduration=60.668811043 podStartE2EDuration="1m0.668811043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:57.667253843 +0000 UTC m=+67.203905044" watchObservedRunningTime="2026-09-11 00:58:57.668811043 +0000 UTC m=+67.205462246"
Sep 11 00:58:57.978854 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:57.978773    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-mrjfb"
Sep 11 00:58:58.584298 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:58.584172    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:58.587480 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:58.587451    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:58.637687 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:58.637664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:58.639065 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:58.639043    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6994df4f88-cwhzx"
Sep 11 00:58:58.639299 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:58.639278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:58.639396 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:58.639366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:58:58.639451 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:58.639423    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:58.639491 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:58:58.639479    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls podName:1c50397a-edd8-48e0-87ba-71e8f359f890 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:14.639462671 +0000 UTC m=+84.176113866 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmm6" (UID: "1c50397a-edd8-48e0-87ba-71e8f359f890") : secret "insights-runtime-extractor-tls" not found
Sep 11 00:58:58.640105 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:58:58.640060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c50397a-edd8-48e0-87ba-71e8f359f890-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:59:00.858356 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:00.858301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:59:00.861322 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:00.861273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614f95d1-510e-4863-ac36-e2aa7a01ccef-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c2z44\" (UID: \"614f95d1-510e-4863-ac36-e2aa7a01ccef\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:59:01.091136 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:01.091097    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hbvts\""
Sep 11 00:59:01.098523 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:01.098499    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"
Sep 11 00:59:02.017917 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.017791    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44"]
Sep 11 00:59:02.654856 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.654816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8" event={"ID":"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18","Type":"ContainerStarted","Data":"4958df2c4c2d9af294756d6fc63d48f05a26b0eb9ca48c6ec98db65b2445b835"}
Sep 11 00:59:02.655044 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.654876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8" event={"ID":"e69cf70a-9cc8-4ddc-820f-cca3e67c2a18","Type":"ContainerStarted","Data":"6d5237934b9fb7994029e15866e9d44635a7407e42bb6670d2e170f9c88633b9"}
Sep 11 00:59:02.656331 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.656293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2" event={"ID":"bde7aa89-7b63-4dfb-b48e-12ea9734ab7d","Type":"ContainerStarted","Data":"83e678d18bc46964e2cb44e77e828a4376cf31ba013b2c3638f0c73e0376fc2e"}
Sep 11 00:59:02.657761 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.657740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m4sbv" event={"ID":"a9227d14-ae58-49a3-9795-626c24256f78","Type":"ContainerStarted","Data":"9a7a3cd8119fea52f8a2368aab595c33fbc151f4922b5036b06a4c7b8943d07e"}
Sep 11 00:59:02.659575 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.659553    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rncnx" event={"ID":"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86","Type":"ContainerStarted","Data":"e0e95b2296fbd9bf50c4cdcc92fb5e0b79115246f6af96d3e2e1aee7de0f329f"}
Sep 11 00:59:02.659676 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.659593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rncnx" event={"ID":"583bbbc2-81a6-4a5f-b224-46ec7e1ecd86","Type":"ContainerStarted","Data":"57398361f1cd8bb5e5e1444f2d72eb19613bdba002ddd44dd191a736aa238c11"}
Sep 11 00:59:02.659676 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.659623    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rncnx"
Sep 11 00:59:02.660773 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.660750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44" event={"ID":"614f95d1-510e-4863-ac36-e2aa7a01ccef","Type":"ContainerStarted","Data":"575db2d024fdf6822220b682f5700769f9adaf5c7406fac62ccf1554169853ed"}
Sep 11 00:59:02.669478 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.669429    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-bxnz8" podStartSLOduration=60.864236020999996 podStartE2EDuration="1m5.669417183s" totalImagesPullingTime="4.805181162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:02.669402018 +0000 UTC m=+72.206053220" watchObservedRunningTime="2026-09-11 00:59:02.669417183 +0000 UTC m=+72.206068383"
Sep 11 00:59:02.685259 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.685217    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m4sbv" podStartSLOduration=65.939432947 podStartE2EDuration="1m11.685202856s" totalImagesPullingTime="5.745769909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:02.684984773 +0000 UTC m=+72.221635977" watchObservedRunningTime="2026-09-11 00:59:02.685202856 +0000 UTC m=+72.221854056"
Sep 11 00:59:02.701329 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.701271    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rncnx" podStartSLOduration=33.891667206 podStartE2EDuration="38.701259727s" totalImagesPullingTime="4.809592521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:02.700536344 +0000 UTC m=+72.237187546" watchObservedRunningTime="2026-09-11 00:59:02.701259727 +0000 UTC m=+72.237910927"
Sep 11 00:59:02.715833 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:02.715784    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-qjmm2" podStartSLOduration=60.792862563 podStartE2EDuration="1m5.715766757s" totalImagesPullingTime="4.922904194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:02.71495329 +0000 UTC m=+72.251604494" watchObservedRunningTime="2026-09-11 00:59:02.715766757 +0000 UTC m=+72.252417959"
Sep 11 00:59:04.669302 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:04.669265    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44" event={"ID":"614f95d1-510e-4863-ac36-e2aa7a01ccef","Type":"ContainerStarted","Data":"58d21ea202ec5121f365350c58e07289a1142e903c6d0e182229dcac36202bda"}
Sep 11 00:59:04.685464 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:04.685415    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c2z44" podStartSLOduration=34.97065138 podStartE2EDuration="36.685400732s" totalImagesPullingTime="1.714749352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:04.683810464 +0000 UTC m=+74.220461664" watchObservedRunningTime="2026-09-11 00:59:04.685400732 +0000 UTC m=+74.222051981"
Sep 11 00:59:12.666465 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:12.666430    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rncnx"
Sep 11 00:59:14.671321 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:14.671187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:59:14.673512 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:14.673495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c50397a-edd8-48e0-87ba-71e8f359f890-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmm6\" (UID: \"1c50397a-edd8-48e0-87ba-71e8f359f890\") " pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:59:14.898521 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:14.898490    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtm2j\""
Sep 11 00:59:14.906298 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:14.906280    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tvmm6"
Sep 11 00:59:15.196011 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:15.195936    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tvmm6"]
Sep 11 00:59:15.198939 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:59:15.198908    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c50397a_edd8_48e0_87ba_71e8f359f890.slice/crio-88cbff18c90592e2a7b3939ffa02b5b5a69bfea2d497ab30d49a75900e8911c4 WatchSource:0}: Error finding container 88cbff18c90592e2a7b3939ffa02b5b5a69bfea2d497ab30d49a75900e8911c4: Status 404 returned error can't find the container with id 88cbff18c90592e2a7b3939ffa02b5b5a69bfea2d497ab30d49a75900e8911c4
Sep 11 00:59:15.711285 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:15.711245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvmm6" event={"ID":"1c50397a-edd8-48e0-87ba-71e8f359f890","Type":"ContainerStarted","Data":"aade026911eb943da7779cdf55005c40309dd0cb93b80bfa36b63f98e9a29fa4"}
Sep 11 00:59:15.711780 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:15.711343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvmm6" event={"ID":"1c50397a-edd8-48e0-87ba-71e8f359f890","Type":"ContainerStarted","Data":"88cbff18c90592e2a7b3939ffa02b5b5a69bfea2d497ab30d49a75900e8911c4"}
Sep 11 00:59:16.473809 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:16.473773    2583 patch_prober.go:28] interesting pod/image-registry-657fb85584-4wbhr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 00:59:16.473969 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:16.473840    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-657fb85584-4wbhr" podUID="191b3b23-de69-4658-a4ca-90bdef9cf7e1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 00:59:16.722763 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:16.722729    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvmm6" event={"ID":"1c50397a-edd8-48e0-87ba-71e8f359f890","Type":"ContainerStarted","Data":"a1ac25ff527a7b365376f5286eaaedaa2493c1bc780600906d7307574181ee42"}
Sep 11 00:59:17.728493 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:17.728456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvmm6" event={"ID":"1c50397a-edd8-48e0-87ba-71e8f359f890","Type":"ContainerStarted","Data":"3577f85855684ad2b7ecf404c3892a2fd9e46ffca0f1fb3e500d80a5f4ffe2ce"}
Sep 11 00:59:17.747317 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:17.747265    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tvmm6" podStartSLOduration=34.096401043 podStartE2EDuration="35.747251357s" totalImagesPullingTime="1.650850314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:58:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:17.746670695 +0000 UTC m=+87.283321907" watchObservedRunningTime="2026-09-11 00:59:17.747251357 +0000 UTC m=+87.283902566"
Sep 11 00:59:18.829662 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:18.829621    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nr6xd"]
Sep 11 00:59:18.832326 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:18.832285    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:18.838031 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:18.838001    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 00:59:18.838265 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:18.838245    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vwq6p\""
Sep 11 00:59:18.838504 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:18.838483    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 00:59:18.838649 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:18.838566    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 00:59:18.838649 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:18.838594    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 00:59:19.008253 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-sys\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.008253 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-tls\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.008478 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-metrics-client-ca\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.008478 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.008478 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-root\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.008478 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-wtmp\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.008639 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4ln\" (UniqueName: \"kubernetes.io/projected/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-kube-api-access-4v4ln\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.008639 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.008639 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.008594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-textfile\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110007 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.109928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110007 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.109964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-root\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110007 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.109988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-wtmp\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110007 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4ln\" (UniqueName: \"kubernetes.io/projected/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-kube-api-access-4v4ln\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110357 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110357 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-root\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110357 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-textfile\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110357 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-sys\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110357 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-tls\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110357 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-wtmp\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110357 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-metrics-client-ca\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110712 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-textfile\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110762 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-sys\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110810 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.110908 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.110886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-metrics-client-ca\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.112429 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.112409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-tls\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.112529 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.112480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.119730 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.119710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4ln\" (UniqueName: \"kubernetes.io/projected/30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5-kube-api-access-4v4ln\") pod \"node-exporter-nr6xd\" (UID: \"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5\") " pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.144771 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.144737    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nr6xd"
Sep 11 00:59:19.155196 ip-10-0-142-82 kubenswrapper[2583]: W0911 00:59:19.155171    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c95fb1_24c5_4f71_8ff7_3fb85b2d9ec5.slice/crio-47ebf21b0a773bf5c5dbccfedf884ca4372349037fdb05b25eef408b6a566a07 WatchSource:0}: Error finding container 47ebf21b0a773bf5c5dbccfedf884ca4372349037fdb05b25eef408b6a566a07: Status 404 returned error can't find the container with id 47ebf21b0a773bf5c5dbccfedf884ca4372349037fdb05b25eef408b6a566a07
Sep 11 00:59:19.646271 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.646240    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:59:19.738614 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:19.738577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nr6xd" event={"ID":"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5","Type":"ContainerStarted","Data":"47ebf21b0a773bf5c5dbccfedf884ca4372349037fdb05b25eef408b6a566a07"}
Sep 11 00:59:20.744026 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:20.743988    2583 generic.go:356] "Generic (PLEG): container finished" podID="30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5" containerID="9e5554b750caea38b67cb2dbfc110a56454503269d7c15338bc488daf2bb043f" exitCode=0
Sep 11 00:59:20.744419 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:20.744076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nr6xd" event={"ID":"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5","Type":"ContainerDied","Data":"9e5554b750caea38b67cb2dbfc110a56454503269d7c15338bc488daf2bb043f"}
Sep 11 00:59:21.749772 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:21.749734    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nr6xd" event={"ID":"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5","Type":"ContainerStarted","Data":"622a49ad7527190de9c3ac2b4b734e449c4b12e41379558dbf3485642dc02fee"}
Sep 11 00:59:21.750167 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:21.749788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nr6xd" event={"ID":"30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5","Type":"ContainerStarted","Data":"ba1b66fbc008d4144434d1f0067976a2837b73782958ae2989c61af32f5def7a"}
Sep 11 00:59:21.787141 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:21.787080    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nr6xd" podStartSLOduration=2.853393556 podStartE2EDuration="3.787065702s" totalImagesPullingTime="869.8724ms" totalInitContainerRuntime="63.799746ms" isStatefulPod=true podCreationTimestamp="2026-09-11 00:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:21.785623468 +0000 UTC m=+91.322274679" watchObservedRunningTime="2026-09-11 00:59:21.787065702 +0000 UTC m=+91.323716903"
Sep 11 00:59:27.606422 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:27.606389    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-657fb85584-4wbhr"]
Sep 11 00:59:27.636576 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:27.636548    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-psmdm"
Sep 11 00:59:47.847191 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:47.847156    2583 generic.go:356] "Generic (PLEG): container finished" podID="27c3cca3-b650-4a88-a6c6-47a747ec800e" containerID="0ccfeb01664f596a578d3d52d7edb0d7b9f868057a0c742ddf122e0da0559202" exitCode=0
Sep 11 00:59:47.847621 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:47.847224    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6" event={"ID":"27c3cca3-b650-4a88-a6c6-47a747ec800e","Type":"ContainerDied","Data":"0ccfeb01664f596a578d3d52d7edb0d7b9f868057a0c742ddf122e0da0559202"}
Sep 11 00:59:47.847621 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:47.847545    2583 scope.go:117] "RemoveContainer" containerID="0ccfeb01664f596a578d3d52d7edb0d7b9f868057a0c742ddf122e0da0559202"
Sep 11 00:59:48.852424 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:48.852391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wfjl6" event={"ID":"27c3cca3-b650-4a88-a6c6-47a747ec800e","Type":"ContainerStarted","Data":"1851ffc40d979a5f6d7b3f2ceaa5e39edf0efd41256be52b4af5d9cd91c7364e"}
Sep 11 00:59:52.625901 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.625829    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-657fb85584-4wbhr" podUID="191b3b23-de69-4658-a4ca-90bdef9cf7e1" containerName="registry" containerID="cri-o://af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274" gracePeriod=30
Sep 11 00:59:52.861982 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.861959    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:59:52.867575 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.867547    2583 generic.go:356] "Generic (PLEG): container finished" podID="191b3b23-de69-4658-a4ca-90bdef9cf7e1" containerID="af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274" exitCode=0
Sep 11 00:59:52.867680 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.867589    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657fb85584-4wbhr"
Sep 11 00:59:52.867680 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.867625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657fb85584-4wbhr" event={"ID":"191b3b23-de69-4658-a4ca-90bdef9cf7e1","Type":"ContainerDied","Data":"af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274"}
Sep 11 00:59:52.867749 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.867704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657fb85584-4wbhr" event={"ID":"191b3b23-de69-4658-a4ca-90bdef9cf7e1","Type":"ContainerDied","Data":"1c92adf51e62e53ba5b6bf37408bc3dd40092a9d3e12430681cb8452d7861f71"}
Sep 11 00:59:52.867749 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.867727    2583 scope.go:117] "RemoveContainer" containerID="af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274"
Sep 11 00:59:52.874993 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.874977    2583 scope.go:117] "RemoveContainer" containerID="af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274"
Sep 11 00:59:52.875239 ip-10-0-142-82 kubenswrapper[2583]: E0911 00:59:52.875221    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274\": container with ID starting with af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274 not found: ID does not exist" containerID="af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274"
Sep 11 00:59:52.875296 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.875248    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274"} err="failed to get container status \"af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274\": rpc error: code = NotFound desc = could not find container \"af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274\": container with ID starting with af26db2e288db71d4e00aa3d9da40648f847ddb8f610e8ab6817ab7c5dbcb274 not found: ID does not exist"
Sep 11 00:59:52.984848 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.984779    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/191b3b23-de69-4658-a4ca-90bdef9cf7e1-ca-trust-extracted\") pod \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") "
Sep 11 00:59:52.984848 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.984819    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-image-registry-private-configuration\") pod \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") "
Sep 11 00:59:52.985016 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.984850    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbmtk\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-kube-api-access-vbmtk\") pod \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") "
Sep 11 00:59:52.985016 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.984882    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-bound-sa-token\") pod \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") "
Sep 11 00:59:52.985016 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.984921    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-trusted-ca\") pod \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") "
Sep 11 00:59:52.985016 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.984953    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") pod \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") "
Sep 11 00:59:52.985016 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.984977    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-installation-pull-secrets\") pod \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") "
Sep 11 00:59:52.985253 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.985029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-certificates\") pod \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\" (UID: \"191b3b23-de69-4658-a4ca-90bdef9cf7e1\") "
Sep 11 00:59:52.985567 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.985410    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "191b3b23-de69-4658-a4ca-90bdef9cf7e1" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:59:52.985708 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.985597    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "191b3b23-de69-4658-a4ca-90bdef9cf7e1" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:59:52.987540 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.987483    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-kube-api-access-vbmtk" (OuterVolumeSpecName: "kube-api-access-vbmtk") pod "191b3b23-de69-4658-a4ca-90bdef9cf7e1" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1"). InnerVolumeSpecName "kube-api-access-vbmtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:59:52.987723 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.987695    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "191b3b23-de69-4658-a4ca-90bdef9cf7e1" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:59:52.987816 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.987726    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "191b3b23-de69-4658-a4ca-90bdef9cf7e1" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:59:52.987816 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.987736    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "191b3b23-de69-4658-a4ca-90bdef9cf7e1" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:59:52.987895 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.987817    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "191b3b23-de69-4658-a4ca-90bdef9cf7e1" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:59:52.992781 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:52.992761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191b3b23-de69-4658-a4ca-90bdef9cf7e1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "191b3b23-de69-4658-a4ca-90bdef9cf7e1" (UID: "191b3b23-de69-4658-a4ca-90bdef9cf7e1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:59:53.085605 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.085583    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-certificates\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 00:59:53.085605 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.085604    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/191b3b23-de69-4658-a4ca-90bdef9cf7e1-ca-trust-extracted\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 00:59:53.085734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.085613    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-image-registry-private-configuration\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 00:59:53.085734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.085623    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbmtk\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-kube-api-access-vbmtk\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 00:59:53.085734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.085632    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-bound-sa-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 00:59:53.085734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.085640    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/191b3b23-de69-4658-a4ca-90bdef9cf7e1-trusted-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 00:59:53.085734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.085647    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/191b3b23-de69-4658-a4ca-90bdef9cf7e1-registry-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 00:59:53.085734 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.085655    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/191b3b23-de69-4658-a4ca-90bdef9cf7e1-installation-pull-secrets\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 00:59:53.182919 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.182890    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-657fb85584-4wbhr"]
Sep 11 00:59:53.188070 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:53.188049    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-657fb85584-4wbhr"]
Sep 11 00:59:54.863815 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:54.863775    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" podUID="0429739a-4188-4eb1-b97a-8f7c2e172a71" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 00:59:55.036246 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:55.036214    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="191b3b23-de69-4658-a4ca-90bdef9cf7e1" path="/var/lib/kubelet/pods/191b3b23-de69-4658-a4ca-90bdef9cf7e1/volumes"
Sep 11 00:59:57.889503 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:57.889470    2583 generic.go:356] "Generic (PLEG): container finished" podID="a10e86f3-6bcd-42da-a580-e218495a161b" containerID="83f5c499226bbf7e3f7717d71deab0bf2353789fdf8931efa42011bb46755776" exitCode=0
Sep 11 00:59:57.889979 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:57.889539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-km5j6" event={"ID":"a10e86f3-6bcd-42da-a580-e218495a161b","Type":"ContainerDied","Data":"83f5c499226bbf7e3f7717d71deab0bf2353789fdf8931efa42011bb46755776"}
Sep 11 00:59:57.889979 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:57.889886    2583 scope.go:117] "RemoveContainer" containerID="83f5c499226bbf7e3f7717d71deab0bf2353789fdf8931efa42011bb46755776"
Sep 11 00:59:57.891205 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:57.891184    2583 generic.go:356] "Generic (PLEG): container finished" podID="cefebf34-9896-4f29-84b3-596e6bbb0b9e" containerID="c09671f895c5f27cfc86408b902bb714b2380178de3a2d1621063a703d0aec91" exitCode=0
Sep 11 00:59:57.891285 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:57.891266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4" event={"ID":"cefebf34-9896-4f29-84b3-596e6bbb0b9e","Type":"ContainerDied","Data":"c09671f895c5f27cfc86408b902bb714b2380178de3a2d1621063a703d0aec91"}
Sep 11 00:59:57.891572 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:57.891556    2583 scope.go:117] "RemoveContainer" containerID="c09671f895c5f27cfc86408b902bb714b2380178de3a2d1621063a703d0aec91"
Sep 11 00:59:58.897385 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:58.897349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-km5j6" event={"ID":"a10e86f3-6bcd-42da-a580-e218495a161b","Type":"ContainerStarted","Data":"f4fec9d6f78b42669d458660f0ab5f5ec92bbb6cf3277123712540a9a3376703"}
Sep 11 00:59:58.899285 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:58.899262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-k9cb4" event={"ID":"cefebf34-9896-4f29-84b3-596e6bbb0b9e","Type":"ContainerStarted","Data":"c7effc1c60f547be1c9c9ff06f191fd6a81a5b8b2e3bb2bec9c4f5a5ac01ab9d"}
Sep 11 00:59:59.737473 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:59.737433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:59:59.739800 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:59.739764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fca14ea-d910-47a0-9563-3ea2ff660cb6-metrics-certs\") pod \"network-metrics-daemon-qrwkr\" (UID: \"2fca14ea-d910-47a0-9563-3ea2ff660cb6\") " pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 00:59:59.950008 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:59.949976    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zpsrm\""
Sep 11 00:59:59.957711 ip-10-0-142-82 kubenswrapper[2583]: I0911 00:59:59.957687    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrwkr"
Sep 11 01:00:00.223405 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:00.223295    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qrwkr"]
Sep 11 01:00:00.226008 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:00:00.225973    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fca14ea_d910_47a0_9563_3ea2ff660cb6.slice/crio-f412c88f85976628e2324877eaa27c7a52f353c11ef29e1f925b74c0e44c71f7 WatchSource:0}: Error finding container f412c88f85976628e2324877eaa27c7a52f353c11ef29e1f925b74c0e44c71f7: Status 404 returned error can't find the container with id f412c88f85976628e2324877eaa27c7a52f353c11ef29e1f925b74c0e44c71f7
Sep 11 01:00:00.909935 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:00.909897    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrwkr" event={"ID":"2fca14ea-d910-47a0-9563-3ea2ff660cb6","Type":"ContainerStarted","Data":"f412c88f85976628e2324877eaa27c7a52f353c11ef29e1f925b74c0e44c71f7"}
Sep 11 01:00:01.915287 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:01.915253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrwkr" event={"ID":"2fca14ea-d910-47a0-9563-3ea2ff660cb6","Type":"ContainerStarted","Data":"71895dae562a3845084d4054113808f65f52d960c0e8c3330f2282a70c45a529"}
Sep 11 01:00:01.915669 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:01.915321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrwkr" event={"ID":"2fca14ea-d910-47a0-9563-3ea2ff660cb6","Type":"ContainerStarted","Data":"cca9f70e6ae11deb6051616a898910a02a11be09cb720f22885ea12f2157472d"}
Sep 11 01:00:01.932521 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:01.932471    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qrwkr" podStartSLOduration=129.887627932 podStartE2EDuration="2m10.932457141s" totalImagesPullingTime="1.044829209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:00:01.931562838 +0000 UTC m=+131.468214240" watchObservedRunningTime="2026-09-11 01:00:01.932457141 +0000 UTC m=+131.469108342"
Sep 11 01:00:04.863165 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:04.863119    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" podUID="0429739a-4188-4eb1-b97a-8f7c2e172a71" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 01:00:14.862923 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:14.862885    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" podUID="0429739a-4188-4eb1-b97a-8f7c2e172a71" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 01:00:14.863406 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:14.862963    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz"
Sep 11 01:00:14.863603 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:14.863580    2583 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2c945c1d5581241ae1b543976f12ae0b7943472327ac0ea379b39fc2121335a9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 11 01:00:14.863682 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:14.863628    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" podUID="0429739a-4188-4eb1-b97a-8f7c2e172a71" containerName="service-proxy" containerID="cri-o://2c945c1d5581241ae1b543976f12ae0b7943472327ac0ea379b39fc2121335a9" gracePeriod=30
Sep 11 01:00:15.973712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:15.973680    2583 generic.go:356] "Generic (PLEG): container finished" podID="0429739a-4188-4eb1-b97a-8f7c2e172a71" containerID="2c945c1d5581241ae1b543976f12ae0b7943472327ac0ea379b39fc2121335a9" exitCode=2
Sep 11 01:00:15.974086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:15.973746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" event={"ID":"0429739a-4188-4eb1-b97a-8f7c2e172a71","Type":"ContainerDied","Data":"2c945c1d5581241ae1b543976f12ae0b7943472327ac0ea379b39fc2121335a9"}
Sep 11 01:00:15.974086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:00:15.973794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cb86697bc-64dwz" event={"ID":"0429739a-4188-4eb1-b97a-8f7c2e172a71","Type":"ContainerStarted","Data":"a1ffaca6c8d3d996f73208b90d3af23083aa11b2c147fb2c4f96ed14ec2d9c16"}
Sep 11 01:02:50.840295 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:02:50.840257    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:02:50.843266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:02:50.843241    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:02:50.846351 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:02:50.846331    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:02:50.849696 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:02:50.849676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:02:50.850631 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:02:50.850612    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 01:04:10.189132 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.189096    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"]
Sep 11 01:04:10.189627 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.189401    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191b3b23-de69-4658-a4ca-90bdef9cf7e1" containerName="registry"
Sep 11 01:04:10.189627 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.189414    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="191b3b23-de69-4658-a4ca-90bdef9cf7e1" containerName="registry"
Sep 11 01:04:10.189627 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.189467    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="191b3b23-de69-4658-a4ca-90bdef9cf7e1" containerName="registry"
Sep 11 01:04:10.192301 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.192286    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.198552 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.198526    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h5l9p\""
Sep 11 01:04:10.198867 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.198847    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 01:04:10.198952 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.198876    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 01:04:10.200357 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.200338    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 01:04:10.200429 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.200337    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 01:04:10.213443 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.213421    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"]
Sep 11 01:04:10.236638 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.236605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-tmp\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.236724 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.236644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.236724 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.236673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-manifests\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.236793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.236730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.236793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.236766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntvxs\" (UniqueName: \"kubernetes.io/projected/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-kube-api-access-ntvxs\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.337846 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.337819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-tmp\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.337991 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.337853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.337991 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.337873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-manifests\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.337991 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.337982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.338129 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.338012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntvxs\" (UniqueName: \"kubernetes.io/projected/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-kube-api-access-ntvxs\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.338379 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.338353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-manifests\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.340256 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.340229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-tmp\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.340417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.340371    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.340417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.340382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.351137 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.351117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntvxs\" (UniqueName: \"kubernetes.io/projected/dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7-kube-api-access-ntvxs\") pod \"opendatahub-operator-controller-manager-8bd47c7d5-jmgzc\" (UID: \"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7\") " pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.502220 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.502134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:10.799115 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.799086    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"]
Sep 11 01:04:10.800556 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:04:10.800532    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd24253c_0f5f_4c69_a4a2_ce3f9e18f4f7.slice/crio-53bb1e0e8d7ecb134c54db4075f0434b38276b117a17bec8c67f796e0cae8149 WatchSource:0}: Error finding container 53bb1e0e8d7ecb134c54db4075f0434b38276b117a17bec8c67f796e0cae8149: Status 404 returned error can't find the container with id 53bb1e0e8d7ecb134c54db4075f0434b38276b117a17bec8c67f796e0cae8149
Sep 11 01:04:10.802467 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.802450    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 01:04:10.844538 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:10.844507    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc" event={"ID":"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7","Type":"ContainerStarted","Data":"53bb1e0e8d7ecb134c54db4075f0434b38276b117a17bec8c67f796e0cae8149"}
Sep 11 01:04:13.859936 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:13.859896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc" event={"ID":"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7","Type":"ContainerStarted","Data":"645c1cfcb4ba3b1b0cb4e6f23f89cedc919226a16cb6f17280d0db47d5c20978"}
Sep 11 01:04:14.865885 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:14.865851    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7" containerID="645c1cfcb4ba3b1b0cb4e6f23f89cedc919226a16cb6f17280d0db47d5c20978" exitCode=0
Sep 11 01:04:14.866272 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:14.865924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc" event={"ID":"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7","Type":"ContainerDied","Data":"645c1cfcb4ba3b1b0cb4e6f23f89cedc919226a16cb6f17280d0db47d5c20978"}
Sep 11 01:04:15.871847 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:15.871811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc" event={"ID":"dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7","Type":"ContainerStarted","Data":"1236f49c3ffa94373eb795815a853823c699478945df0023bc0e97c7bb7311f1"}
Sep 11 01:04:15.872325 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:15.871936    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:15.895667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:15.895622    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc" podStartSLOduration=1.590608043 podStartE2EDuration="5.895606544s" totalImagesPullingTime="3.234640912s" totalInitContainerRuntime="1.070357589s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:04:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:04:15.894031384 +0000 UTC m=+385.430682584" watchObservedRunningTime="2026-09-11 01:04:15.895606544 +0000 UTC m=+385.432257745"
Sep 11 01:04:26.877746 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:26.877712    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8bd47c7d5-jmgzc"
Sep 11 01:04:40.294147 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.294114    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rz5pw"]
Sep 11 01:04:40.303833 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.303808    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:40.306491 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.306468    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2x4xg\""
Sep 11 01:04:40.306739 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.306713    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 01:04:40.312838 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.312814    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rz5pw"]
Sep 11 01:04:40.371012 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.370979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7zm\" (UniqueName: \"kubernetes.io/projected/5dd9fca6-85c2-434d-a586-8c0c68bd5143-kube-api-access-4d7zm\") pod \"odh-model-controller-7998c67dc4-rz5pw\" (UID: \"5dd9fca6-85c2-434d-a586-8c0c68bd5143\") " pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:40.371161 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.371044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dd9fca6-85c2-434d-a586-8c0c68bd5143-cert\") pod \"odh-model-controller-7998c67dc4-rz5pw\" (UID: \"5dd9fca6-85c2-434d-a586-8c0c68bd5143\") " pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:40.472337 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.472264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dd9fca6-85c2-434d-a586-8c0c68bd5143-cert\") pod \"odh-model-controller-7998c67dc4-rz5pw\" (UID: \"5dd9fca6-85c2-434d-a586-8c0c68bd5143\") " pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:40.472522 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.472354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7zm\" (UniqueName: \"kubernetes.io/projected/5dd9fca6-85c2-434d-a586-8c0c68bd5143-kube-api-access-4d7zm\") pod \"odh-model-controller-7998c67dc4-rz5pw\" (UID: \"5dd9fca6-85c2-434d-a586-8c0c68bd5143\") " pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:40.472522 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:04:40.472441    2583 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 01:04:40.472522 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:04:40.472517    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dd9fca6-85c2-434d-a586-8c0c68bd5143-cert podName:5dd9fca6-85c2-434d-a586-8c0c68bd5143 nodeName:}" failed. No retries permitted until 2026-09-11 01:04:40.972497699 +0000 UTC m=+410.509148883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5dd9fca6-85c2-434d-a586-8c0c68bd5143-cert") pod "odh-model-controller-7998c67dc4-rz5pw" (UID: "5dd9fca6-85c2-434d-a586-8c0c68bd5143") : secret "odh-model-controller-webhook-cert" not found
Sep 11 01:04:40.489631 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.489590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7zm\" (UniqueName: \"kubernetes.io/projected/5dd9fca6-85c2-434d-a586-8c0c68bd5143-kube-api-access-4d7zm\") pod \"odh-model-controller-7998c67dc4-rz5pw\" (UID: \"5dd9fca6-85c2-434d-a586-8c0c68bd5143\") " pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:40.698617 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.698586    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7z9xk"]
Sep 11 01:04:40.700921 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.700902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:40.706400 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.706382    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 01:04:40.716232 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.716209    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7z9xk"]
Sep 11 01:04:40.774526 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.774493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hv7w\" (UniqueName: \"kubernetes.io/projected/f67d2b7e-3262-4cca-8b5f-538ce246dc6e-kube-api-access-5hv7w\") pod \"model-serving-api-54d89bd5fc-7z9xk\" (UID: \"f67d2b7e-3262-4cca-8b5f-538ce246dc6e\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:40.774699 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.774532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67d2b7e-3262-4cca-8b5f-538ce246dc6e-tls-certs\") pod \"model-serving-api-54d89bd5fc-7z9xk\" (UID: \"f67d2b7e-3262-4cca-8b5f-538ce246dc6e\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:40.874911 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.874877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hv7w\" (UniqueName: \"kubernetes.io/projected/f67d2b7e-3262-4cca-8b5f-538ce246dc6e-kube-api-access-5hv7w\") pod \"model-serving-api-54d89bd5fc-7z9xk\" (UID: \"f67d2b7e-3262-4cca-8b5f-538ce246dc6e\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:40.874911 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.874917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67d2b7e-3262-4cca-8b5f-538ce246dc6e-tls-certs\") pod \"model-serving-api-54d89bd5fc-7z9xk\" (UID: \"f67d2b7e-3262-4cca-8b5f-538ce246dc6e\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:40.877480 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.877456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67d2b7e-3262-4cca-8b5f-538ce246dc6e-tls-certs\") pod \"model-serving-api-54d89bd5fc-7z9xk\" (UID: \"f67d2b7e-3262-4cca-8b5f-538ce246dc6e\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:40.883117 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.883094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hv7w\" (UniqueName: \"kubernetes.io/projected/f67d2b7e-3262-4cca-8b5f-538ce246dc6e-kube-api-access-5hv7w\") pod \"model-serving-api-54d89bd5fc-7z9xk\" (UID: \"f67d2b7e-3262-4cca-8b5f-538ce246dc6e\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:40.975473 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.975388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dd9fca6-85c2-434d-a586-8c0c68bd5143-cert\") pod \"odh-model-controller-7998c67dc4-rz5pw\" (UID: \"5dd9fca6-85c2-434d-a586-8c0c68bd5143\") " pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:40.978086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:40.978060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dd9fca6-85c2-434d-a586-8c0c68bd5143-cert\") pod \"odh-model-controller-7998c67dc4-rz5pw\" (UID: \"5dd9fca6-85c2-434d-a586-8c0c68bd5143\") " pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:41.011171 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:41.011148    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:41.215712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:41.215668    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:41.314596 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:41.314572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7z9xk"]
Sep 11 01:04:41.316378 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:04:41.316352    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67d2b7e_3262_4cca_8b5f_538ce246dc6e.slice/crio-c16b0c1a58e8236d58fae9bd5101a88fb1fe2d33fd97c87f182b492e8b5745eb WatchSource:0}: Error finding container c16b0c1a58e8236d58fae9bd5101a88fb1fe2d33fd97c87f182b492e8b5745eb: Status 404 returned error can't find the container with id c16b0c1a58e8236d58fae9bd5101a88fb1fe2d33fd97c87f182b492e8b5745eb
Sep 11 01:04:41.488814 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:41.488743    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rz5pw"]
Sep 11 01:04:41.492023 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:04:41.491999    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd9fca6_85c2_434d_a586_8c0c68bd5143.slice/crio-91a6ca18ff350dae658448d83eaac1799b887906f659870fd319082447a5fa42 WatchSource:0}: Error finding container 91a6ca18ff350dae658448d83eaac1799b887906f659870fd319082447a5fa42: Status 404 returned error can't find the container with id 91a6ca18ff350dae658448d83eaac1799b887906f659870fd319082447a5fa42
Sep 11 01:04:41.986771 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:41.986738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw" event={"ID":"5dd9fca6-85c2-434d-a586-8c0c68bd5143","Type":"ContainerStarted","Data":"91a6ca18ff350dae658448d83eaac1799b887906f659870fd319082447a5fa42"}
Sep 11 01:04:41.988104 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:41.988075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk" event={"ID":"f67d2b7e-3262-4cca-8b5f-538ce246dc6e","Type":"ContainerStarted","Data":"c16b0c1a58e8236d58fae9bd5101a88fb1fe2d33fd97c87f182b492e8b5745eb"}
Sep 11 01:04:46.012451 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:46.012409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw" event={"ID":"5dd9fca6-85c2-434d-a586-8c0c68bd5143","Type":"ContainerStarted","Data":"99a66fe170b58a03241366c0403f93efcd3bd5a425ba812608bd8beec482a904"}
Sep 11 01:04:46.012885 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:46.012511    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:46.013983 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:46.013958    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk" event={"ID":"f67d2b7e-3262-4cca-8b5f-538ce246dc6e","Type":"ContainerStarted","Data":"86796fefb1561c1c1476cac74d8cb48b04c6d381e56e001641bcab5215f03eb1"}
Sep 11 01:04:46.014067 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:46.014014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:04:46.029558 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:46.029508    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw" podStartSLOduration=2.288077562 podStartE2EDuration="6.029490869s" totalImagesPullingTime="3.741413307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:04:46.028153268 +0000 UTC m=+415.564804473" watchObservedRunningTime="2026-09-11 01:04:46.029490869 +0000 UTC m=+415.566142072"
Sep 11 01:04:46.043417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:46.043165    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk" podStartSLOduration=2.170924362 podStartE2EDuration="6.043146783s" totalImagesPullingTime="3.872222421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:04:46.043115942 +0000 UTC m=+415.579767144" watchObservedRunningTime="2026-09-11 01:04:46.043146783 +0000 UTC m=+415.579798011"
Sep 11 01:04:48.664349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.664291    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"]
Sep 11 01:04:48.667873 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.667850    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.671199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.671173    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 01:04:48.671337 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.671198    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 01:04:48.671337 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.671208    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 01:04:48.671542 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.671525    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5bnzz\""
Sep 11 01:04:48.671966 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.671946    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 01:04:48.671966 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.671960    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 01:04:48.686358 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.686338    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"]
Sep 11 01:04:48.742699 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.742670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7555119-b990-429c-a16e-051c6f30b097-cert\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.742842 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.742725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7555119-b990-429c-a16e-051c6f30b097-metrics-cert\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.742842 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.742784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7555119-b990-429c-a16e-051c6f30b097-manager-config\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.742952 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.742863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8w4\" (UniqueName: \"kubernetes.io/projected/d7555119-b990-429c-a16e-051c6f30b097-kube-api-access-jt8w4\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.843943 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.843914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8w4\" (UniqueName: \"kubernetes.io/projected/d7555119-b990-429c-a16e-051c6f30b097-kube-api-access-jt8w4\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.844100 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.843954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7555119-b990-429c-a16e-051c6f30b097-cert\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.844100 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.843990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7555119-b990-429c-a16e-051c6f30b097-metrics-cert\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.844100 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.844008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7555119-b990-429c-a16e-051c6f30b097-manager-config\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.844644 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.844621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7555119-b990-429c-a16e-051c6f30b097-manager-config\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.846417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.846390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7555119-b990-429c-a16e-051c6f30b097-cert\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.846510 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.846493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7555119-b990-429c-a16e-051c6f30b097-metrics-cert\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.852856 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.852836    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8w4\" (UniqueName: \"kubernetes.io/projected/d7555119-b990-429c-a16e-051c6f30b097-kube-api-access-jt8w4\") pod \"lws-controller-manager-5bdf494464-gld47\" (UID: \"d7555119-b990-429c-a16e-051c6f30b097\") " pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:48.977530 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:48.977462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:49.275241 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:49.275139    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"]
Sep 11 01:04:49.277797 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:04:49.277761    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7555119_b990_429c_a16e_051c6f30b097.slice/crio-1fdf2665065a576300f7a3b6cf72a0447fcbed1052f6063aa617e921cd2eba8d WatchSource:0}: Error finding container 1fdf2665065a576300f7a3b6cf72a0447fcbed1052f6063aa617e921cd2eba8d: Status 404 returned error can't find the container with id 1fdf2665065a576300f7a3b6cf72a0447fcbed1052f6063aa617e921cd2eba8d
Sep 11 01:04:50.044115 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:50.043692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47" event={"ID":"d7555119-b990-429c-a16e-051c6f30b097","Type":"ContainerStarted","Data":"1fdf2665065a576300f7a3b6cf72a0447fcbed1052f6063aa617e921cd2eba8d"}
Sep 11 01:04:51.130572 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.130506    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"]
Sep 11 01:04:51.143950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.143903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:51.147094 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.147035    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 01:04:51.147743 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.147532    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hn9rq\""
Sep 11 01:04:51.166807 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.166566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"]
Sep 11 01:04:51.267861 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.267823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghfn\" (UniqueName: \"kubernetes.io/projected/d283a886-ef93-4943-b2e2-650c58a45ae0-kube-api-access-sghfn\") pod \"llmisvc-controller-manager-777c6bd7d5-bz9m2\" (UID: \"d283a886-ef93-4943-b2e2-650c58a45ae0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:51.268026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.267880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d283a886-ef93-4943-b2e2-650c58a45ae0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bz9m2\" (UID: \"d283a886-ef93-4943-b2e2-650c58a45ae0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:51.369407 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.369363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghfn\" (UniqueName: \"kubernetes.io/projected/d283a886-ef93-4943-b2e2-650c58a45ae0-kube-api-access-sghfn\") pod \"llmisvc-controller-manager-777c6bd7d5-bz9m2\" (UID: \"d283a886-ef93-4943-b2e2-650c58a45ae0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:51.369595 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.369436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d283a886-ef93-4943-b2e2-650c58a45ae0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bz9m2\" (UID: \"d283a886-ef93-4943-b2e2-650c58a45ae0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:51.372336 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.372296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d283a886-ef93-4943-b2e2-650c58a45ae0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bz9m2\" (UID: \"d283a886-ef93-4943-b2e2-650c58a45ae0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:51.378589 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.378559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghfn\" (UniqueName: \"kubernetes.io/projected/d283a886-ef93-4943-b2e2-650c58a45ae0-kube-api-access-sghfn\") pod \"llmisvc-controller-manager-777c6bd7d5-bz9m2\" (UID: \"d283a886-ef93-4943-b2e2-650c58a45ae0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:51.455611 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.455534    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:51.881921 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:51.881816    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"]
Sep 11 01:04:51.884410 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:04:51.884386    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd283a886_ef93_4943_b2e2_650c58a45ae0.slice/crio-0196914c9866d7a4f6f6696d81ac9ae40556ca5ba11c476dc799c780d257936d WatchSource:0}: Error finding container 0196914c9866d7a4f6f6696d81ac9ae40556ca5ba11c476dc799c780d257936d: Status 404 returned error can't find the container with id 0196914c9866d7a4f6f6696d81ac9ae40556ca5ba11c476dc799c780d257936d
Sep 11 01:04:52.052901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:52.052819    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2" event={"ID":"d283a886-ef93-4943-b2e2-650c58a45ae0","Type":"ContainerStarted","Data":"0196914c9866d7a4f6f6696d81ac9ae40556ca5ba11c476dc799c780d257936d"}
Sep 11 01:04:52.054218 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:52.054191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47" event={"ID":"d7555119-b990-429c-a16e-051c6f30b097","Type":"ContainerStarted","Data":"618f6eaafcd4bf604bb8f71c11fde30872e045357e0dff4249fc41430789f119"}
Sep 11 01:04:52.054353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:52.054336    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:04:52.077070 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:52.077030    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47" podStartSLOduration=1.690003633 podStartE2EDuration="4.077015373s" totalImagesPullingTime="2.38701174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:04:52.074661391 +0000 UTC m=+421.611312593" watchObservedRunningTime="2026-09-11 01:04:52.077015373 +0000 UTC m=+421.613666642"
Sep 11 01:04:55.071450 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:55.071416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2" event={"ID":"d283a886-ef93-4943-b2e2-650c58a45ae0","Type":"ContainerStarted","Data":"6ef6e3700be9960fb6abe2cf5fabdbb11815292596ed941ded079117b82c3c2d"}
Sep 11 01:04:55.071819 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:55.071484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:04:55.090216 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:55.090170    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2" podStartSLOduration=1.749964935 podStartE2EDuration="4.090153788s" totalImagesPullingTime="2.340188853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:04:55.089213127 +0000 UTC m=+424.625864328" watchObservedRunningTime="2026-09-11 01:04:55.090153788 +0000 UTC m=+424.626804989"
Sep 11 01:04:57.019888 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:57.019849    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rz5pw"
Sep 11 01:04:57.023040 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:04:57.023013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7z9xk"
Sep 11 01:05:03.062792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:05:03.062766    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bdf494464-gld47"
Sep 11 01:05:26.078122 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:05:26.078048    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bz9m2"
Sep 11 01:06:07.400975 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.400943    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"]
Sep 11 01:06:07.403147 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.403131    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:07.405760 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.405740    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 01:06:07.405852 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.405766    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 01:06:07.405931 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.405914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fb2j7\""
Sep 11 01:06:07.418266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.418244    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"]
Sep 11 01:06:07.475145 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.475113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspc9\" (UniqueName: \"kubernetes.io/projected/50220bad-8fd5-450c-9400-e36969e1aac0-kube-api-access-fspc9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" (UID: \"50220bad-8fd5-450c-9400-e36969e1aac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:07.475293 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.475160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50220bad-8fd5-450c-9400-e36969e1aac0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" (UID: \"50220bad-8fd5-450c-9400-e36969e1aac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:07.576265 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.576231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspc9\" (UniqueName: \"kubernetes.io/projected/50220bad-8fd5-450c-9400-e36969e1aac0-kube-api-access-fspc9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" (UID: \"50220bad-8fd5-450c-9400-e36969e1aac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:07.576440 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.576283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50220bad-8fd5-450c-9400-e36969e1aac0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" (UID: \"50220bad-8fd5-450c-9400-e36969e1aac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:07.576691 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.576674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50220bad-8fd5-450c-9400-e36969e1aac0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" (UID: \"50220bad-8fd5-450c-9400-e36969e1aac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:07.600637 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.600614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspc9\" (UniqueName: \"kubernetes.io/projected/50220bad-8fd5-450c-9400-e36969e1aac0-kube-api-access-fspc9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" (UID: \"50220bad-8fd5-450c-9400-e36969e1aac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:07.713616 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.713542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:07.991193 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:07.991115    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"]
Sep 11 01:06:07.994588 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:06:07.994560    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50220bad_8fd5_450c_9400_e36969e1aac0.slice/crio-2c7aef49d6a73ad20f4d9afa96b22018086d8241540b9934ebc8b0c8508b9b11 WatchSource:0}: Error finding container 2c7aef49d6a73ad20f4d9afa96b22018086d8241540b9934ebc8b0c8508b9b11: Status 404 returned error can't find the container with id 2c7aef49d6a73ad20f4d9afa96b22018086d8241540b9934ebc8b0c8508b9b11
Sep 11 01:06:08.383516 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:08.383480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht" event={"ID":"50220bad-8fd5-450c-9400-e36969e1aac0","Type":"ContainerStarted","Data":"2c7aef49d6a73ad20f4d9afa96b22018086d8241540b9934ebc8b0c8508b9b11"}
Sep 11 01:06:12.407780 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:12.407745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht" event={"ID":"50220bad-8fd5-450c-9400-e36969e1aac0","Type":"ContainerStarted","Data":"6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa"}
Sep 11 01:06:12.408159 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:12.407835    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:12.430182 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:12.430135    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht" podStartSLOduration=1.585322592 podStartE2EDuration="5.430119642s" totalImagesPullingTime="3.84479705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:12.42758865 +0000 UTC m=+501.964239851" watchObservedRunningTime="2026-09-11 01:06:12.430119642 +0000 UTC m=+501.966770843"
Sep 11 01:06:23.414771 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:23.414739    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:24.711465 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.711430    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"]
Sep 11 01:06:24.713565 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.713547    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:24.732973 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.732946    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"]
Sep 11 01:06:24.801170 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.801135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" (UID: \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:24.801292 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.801268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgxv\" (UniqueName: \"kubernetes.io/projected/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-kube-api-access-gmgxv\") pod \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" (UID: \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:24.902039 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.902007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgxv\" (UniqueName: \"kubernetes.io/projected/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-kube-api-access-gmgxv\") pod \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" (UID: \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:24.902186 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.902050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" (UID: \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:24.902404 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.902386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" (UID: \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:24.910942 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:24.910911    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgxv\" (UniqueName: \"kubernetes.io/projected/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-kube-api-access-gmgxv\") pod \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" (UID: \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:25.023278 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.023200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:25.297227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.297200    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"]
Sep 11 01:06:25.299866 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:06:25.299833    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6c2e2e_05c9_4d81_85c3_c1daad43d1f3.slice/crio-8e47aec9c496cd271609648e15ccfe59d877f2684ef3e57ac94c3a744ade80b2 WatchSource:0}: Error finding container 8e47aec9c496cd271609648e15ccfe59d877f2684ef3e57ac94c3a744ade80b2: Status 404 returned error can't find the container with id 8e47aec9c496cd271609648e15ccfe59d877f2684ef3e57ac94c3a744ade80b2
Sep 11 01:06:25.466382 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.466346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs" event={"ID":"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3","Type":"ContainerStarted","Data":"5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7"}
Sep 11 01:06:25.466552 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.466406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs" event={"ID":"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3","Type":"ContainerStarted","Data":"8e47aec9c496cd271609648e15ccfe59d877f2684ef3e57ac94c3a744ade80b2"}
Sep 11 01:06:25.466552 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.466512    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:25.486208 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.486157    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs" podStartSLOduration=1.486143412 podStartE2EDuration="1.486143412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:25.484191509 +0000 UTC m=+515.020842722" watchObservedRunningTime="2026-09-11 01:06:25.486143412 +0000 UTC m=+515.022794613"
Sep 11 01:06:25.597967 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.597939    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"]
Sep 11 01:06:25.606823 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.606794    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"]
Sep 11 01:06:25.617951 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.617898    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"]
Sep 11 01:06:25.618372 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.618343    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" containerName="manager" containerID="cri-o://6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa" gracePeriod=2
Sep 11 01:06:25.619602 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.619580    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"]
Sep 11 01:06:25.619974 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.619958    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" containerName="manager"
Sep 11 01:06:25.620070 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.619977    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" containerName="manager"
Sep 11 01:06:25.620126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.620098    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" containerName="manager"
Sep 11 01:06:25.622417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.622204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:25.628055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.627995    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"]
Sep 11 01:06:25.636525 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:25.636484    2583 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"50220bad-8fd5-450c-9400-e36969e1aac0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-fmpht\": pods \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:25.638526 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:25.638492    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:25.639642 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.639612    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"]
Sep 11 01:06:25.645418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.645398    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"]
Sep 11 01:06:25.645733 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.645718    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" containerName="manager"
Sep 11 01:06:25.645803 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.645738    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" containerName="manager"
Sep 11 01:06:25.645860 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.645850    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" containerName="manager"
Sep 11 01:06:25.647587 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.647572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:25.660457 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:25.660426    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:25.664850 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.664827    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"]
Sep 11 01:06:25.707698 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.707670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ns2\" (UniqueName: \"kubernetes.io/projected/0b515ad9-4d62-473c-ac35-a7364ef967e0-kube-api-access-s6ns2\") pod \"kuadrant-operator-controller-manager-5f685c496f-5vhjs\" (UID: \"0b515ad9-4d62-473c-ac35-a7364ef967e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:25.707841 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.707775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b515ad9-4d62-473c-ac35-a7364ef967e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-5vhjs\" (UID: \"0b515ad9-4d62-473c-ac35-a7364ef967e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:25.707841 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.707820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11b6638d-2c7d-4360-b7f1-685edd3c0589-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nmbqg\" (UID: \"11b6638d-2c7d-4360-b7f1-685edd3c0589\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:25.707943 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.707857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92p4t\" (UniqueName: \"kubernetes.io/projected/11b6638d-2c7d-4360-b7f1-685edd3c0589-kube-api-access-92p4t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nmbqg\" (UID: \"11b6638d-2c7d-4360-b7f1-685edd3c0589\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:25.808581 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.808549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11b6638d-2c7d-4360-b7f1-685edd3c0589-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nmbqg\" (UID: \"11b6638d-2c7d-4360-b7f1-685edd3c0589\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:25.808954 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.808603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92p4t\" (UniqueName: \"kubernetes.io/projected/11b6638d-2c7d-4360-b7f1-685edd3c0589-kube-api-access-92p4t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nmbqg\" (UID: \"11b6638d-2c7d-4360-b7f1-685edd3c0589\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:25.808954 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.808643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ns2\" (UniqueName: \"kubernetes.io/projected/0b515ad9-4d62-473c-ac35-a7364ef967e0-kube-api-access-s6ns2\") pod \"kuadrant-operator-controller-manager-5f685c496f-5vhjs\" (UID: \"0b515ad9-4d62-473c-ac35-a7364ef967e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:25.808954 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.808754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b515ad9-4d62-473c-ac35-a7364ef967e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-5vhjs\" (UID: \"0b515ad9-4d62-473c-ac35-a7364ef967e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:25.809070 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.809051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11b6638d-2c7d-4360-b7f1-685edd3c0589-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nmbqg\" (UID: \"11b6638d-2c7d-4360-b7f1-685edd3c0589\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:25.809114 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.809078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b515ad9-4d62-473c-ac35-a7364ef967e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-5vhjs\" (UID: \"0b515ad9-4d62-473c-ac35-a7364ef967e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:25.816769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.816745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92p4t\" (UniqueName: \"kubernetes.io/projected/11b6638d-2c7d-4360-b7f1-685edd3c0589-kube-api-access-92p4t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nmbqg\" (UID: \"11b6638d-2c7d-4360-b7f1-685edd3c0589\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:25.816898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.816879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ns2\" (UniqueName: \"kubernetes.io/projected/0b515ad9-4d62-473c-ac35-a7364ef967e0-kube-api-access-s6ns2\") pod \"kuadrant-operator-controller-manager-5f685c496f-5vhjs\" (UID: \"0b515ad9-4d62-473c-ac35-a7364ef967e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:25.850050 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.849993    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:25.851971 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:25.851945    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:25.909791 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.909762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fspc9\" (UniqueName: \"kubernetes.io/projected/50220bad-8fd5-450c-9400-e36969e1aac0-kube-api-access-fspc9\") pod \"50220bad-8fd5-450c-9400-e36969e1aac0\" (UID: \"50220bad-8fd5-450c-9400-e36969e1aac0\") "
Sep 11 01:06:25.909916 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.909853    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50220bad-8fd5-450c-9400-e36969e1aac0-extensions-socket-volume\") pod \"50220bad-8fd5-450c-9400-e36969e1aac0\" (UID: \"50220bad-8fd5-450c-9400-e36969e1aac0\") "
Sep 11 01:06:25.910299 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.910270    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50220bad-8fd5-450c-9400-e36969e1aac0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "50220bad-8fd5-450c-9400-e36969e1aac0" (UID: "50220bad-8fd5-450c-9400-e36969e1aac0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:06:25.911965 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.911942    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50220bad-8fd5-450c-9400-e36969e1aac0-kube-api-access-fspc9" (OuterVolumeSpecName: "kube-api-access-fspc9") pod "50220bad-8fd5-450c-9400-e36969e1aac0" (UID: "50220bad-8fd5-450c-9400-e36969e1aac0"). InnerVolumeSpecName "kube-api-access-fspc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:06:25.938100 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.938072    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:25.996651 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:25.996591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:26.010955 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.010932    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fspc9\" (UniqueName: \"kubernetes.io/projected/50220bad-8fd5-450c-9400-e36969e1aac0-kube-api-access-fspc9\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:06:26.010955 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.010954    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50220bad-8fd5-450c-9400-e36969e1aac0-extensions-socket-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:06:26.225501 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.224957    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"]
Sep 11 01:06:26.228048 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:06:26.228017    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b515ad9_4d62_473c_ac35_a7364ef967e0.slice/crio-d513c111f8efb9c729303e798208970e2c524fee58a61813440adfd06acf149e WatchSource:0}: Error finding container d513c111f8efb9c729303e798208970e2c524fee58a61813440adfd06acf149e: Status 404 returned error can't find the container with id d513c111f8efb9c729303e798208970e2c524fee58a61813440adfd06acf149e
Sep 11 01:06:26.289801 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.289779    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"]
Sep 11 01:06:26.291709 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:06:26.291684    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b6638d_2c7d_4360_b7f1_685edd3c0589.slice/crio-6f5fc90b8246d3bda26eac0e0211554eecaff8493507d45493f0d24f63daaf21 WatchSource:0}: Error finding container 6f5fc90b8246d3bda26eac0e0211554eecaff8493507d45493f0d24f63daaf21: Status 404 returned error can't find the container with id 6f5fc90b8246d3bda26eac0e0211554eecaff8493507d45493f0d24f63daaf21
Sep 11 01:06:26.472267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.472227    2583 generic.go:356] "Generic (PLEG): container finished" podID="50220bad-8fd5-450c-9400-e36969e1aac0" containerID="6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa" exitCode=0
Sep 11 01:06:26.472445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.472271    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:26.472445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.472337    2583 scope.go:117] "RemoveContainer" containerID="6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa"
Sep 11 01:06:26.474343 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.474290    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg" event={"ID":"11b6638d-2c7d-4360-b7f1-685edd3c0589","Type":"ContainerStarted","Data":"4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f"}
Sep 11 01:06:26.474462 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.474359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg" event={"ID":"11b6638d-2c7d-4360-b7f1-685edd3c0589","Type":"ContainerStarted","Data":"6f5fc90b8246d3bda26eac0e0211554eecaff8493507d45493f0d24f63daaf21"}
Sep 11 01:06:26.474462 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.474394    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:26.474669 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:26.474643    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:26.476384 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:26.476301    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:26.476494 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.476427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs" event={"ID":"0b515ad9-4d62-473c-ac35-a7364ef967e0","Type":"ContainerStarted","Data":"a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737"}
Sep 11 01:06:26.476557 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.476529    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:26.476611 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.476568    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs" event={"ID":"0b515ad9-4d62-473c-ac35-a7364ef967e0","Type":"ContainerStarted","Data":"d513c111f8efb9c729303e798208970e2c524fee58a61813440adfd06acf149e"}
Sep 11 01:06:26.476755 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:26.476738    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs" container="manager"
Sep 11 01:06:26.481954 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.481938    2583 scope.go:117] "RemoveContainer" containerID="6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa"
Sep 11 01:06:26.482191 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:26.482172    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa\": container with ID starting with 6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa not found: ID does not exist" containerID="6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa"
Sep 11 01:06:26.482240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.482196    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa"} err="failed to get container status \"6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa\": rpc error: code = NotFound desc = could not find container \"6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa\": container with ID starting with 6c772db63f2e387c1a56f6a8c1806e798023323761b287eace3aa810b2cb23fa not found: ID does not exist"
Sep 11 01:06:26.495080 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.494648    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg" podStartSLOduration=1.49463685 podStartE2EDuration="1.49463685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:26.493131667 +0000 UTC m=+516.029782867" watchObservedRunningTime="2026-09-11 01:06:26.49463685 +0000 UTC m=+516.031288050"
Sep 11 01:06:26.495436 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:26.495414    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fmpht\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fmpht"
Sep 11 01:06:26.497111 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:26.497083    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:26.524152 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:26.524101    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs" podStartSLOduration=1.524083199 podStartE2EDuration="1.524083199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:26.522385355 +0000 UTC m=+516.059036558" watchObservedRunningTime="2026-09-11 01:06:26.524083199 +0000 UTC m=+516.060734403"
Sep 11 01:06:27.042828 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.042787    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50220bad-8fd5-450c-9400-e36969e1aac0" path="/var/lib/kubelet/pods/50220bad-8fd5-450c-9400-e36969e1aac0/volumes"
Sep 11 01:06:27.484270 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.484228    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs" podUID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" containerName="manager" containerID="cri-o://5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7" gracePeriod=2
Sep 11 01:06:27.719033 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.719010    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:27.827222 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.827193    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmgxv\" (UniqueName: \"kubernetes.io/projected/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-kube-api-access-gmgxv\") pod \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\" (UID: \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\") "
Sep 11 01:06:27.827375 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.827293    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-extensions-socket-volume\") pod \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\" (UID: \"4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3\") "
Sep 11 01:06:27.827684 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.827653    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" (UID: "4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:06:27.829203 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.829167    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-kube-api-access-gmgxv" (OuterVolumeSpecName: "kube-api-access-gmgxv") pod "4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" (UID: "4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3"). InnerVolumeSpecName "kube-api-access-gmgxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:06:27.928130 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.928092    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-extensions-socket-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:06:27.928130 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:27.928126    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgxv\" (UniqueName: \"kubernetes.io/projected/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3-kube-api-access-gmgxv\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:06:28.490556 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:28.490523    2583 generic.go:356] "Generic (PLEG): container finished" podID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" containerID="5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7" exitCode=0
Sep 11 01:06:28.490977 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:28.490566    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:28.490977 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:28.490573    2583 scope.go:117] "RemoveContainer" containerID="5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7"
Sep 11 01:06:28.494265 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:28.494237    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:28.501207 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:28.501188    2583 scope.go:117] "RemoveContainer" containerID="5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7"
Sep 11 01:06:28.501758 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:28.501736    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7\": container with ID starting with 5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7 not found: ID does not exist" containerID="5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7"
Sep 11 01:06:28.501847 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:28.501763    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7"} err="failed to get container status \"5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7\": rpc error: code = NotFound desc = could not find container \"5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7\": container with ID starting with 5f4a4117b2f29ca4cbb39955a9e2cba6c59cfd4d841b9cc337569bca3de78fe7 not found: ID does not exist"
Sep 11 01:06:28.504634 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:28.504610    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-55mqs\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-55mqs"
Sep 11 01:06:29.036888 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:29.036859    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3" path="/var/lib/kubelet/pods/4a6c2e2e-05c9-4d81-85c3-c1daad43d1f3/volumes"
Sep 11 01:06:37.486479 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.486447    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:37.486899 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.486520    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:37.559454 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.559424    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"]
Sep 11 01:06:37.559625 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.559604    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg" podUID="11b6638d-2c7d-4360-b7f1-685edd3c0589" containerName="manager" containerID="cri-o://4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f" gracePeriod=10
Sep 11 01:06:37.801896 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.801874    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:37.910073 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.910043    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11b6638d-2c7d-4360-b7f1-685edd3c0589-extensions-socket-volume\") pod \"11b6638d-2c7d-4360-b7f1-685edd3c0589\" (UID: \"11b6638d-2c7d-4360-b7f1-685edd3c0589\") "
Sep 11 01:06:37.910073 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.910076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92p4t\" (UniqueName: \"kubernetes.io/projected/11b6638d-2c7d-4360-b7f1-685edd3c0589-kube-api-access-92p4t\") pod \"11b6638d-2c7d-4360-b7f1-685edd3c0589\" (UID: \"11b6638d-2c7d-4360-b7f1-685edd3c0589\") "
Sep 11 01:06:37.910498 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.910475    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b6638d-2c7d-4360-b7f1-685edd3c0589-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "11b6638d-2c7d-4360-b7f1-685edd3c0589" (UID: "11b6638d-2c7d-4360-b7f1-685edd3c0589"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:06:37.912030 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.912002    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b6638d-2c7d-4360-b7f1-685edd3c0589-kube-api-access-92p4t" (OuterVolumeSpecName: "kube-api-access-92p4t") pod "11b6638d-2c7d-4360-b7f1-685edd3c0589" (UID: "11b6638d-2c7d-4360-b7f1-685edd3c0589"). InnerVolumeSpecName "kube-api-access-92p4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:06:37.916743 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.916722    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"]
Sep 11 01:06:37.917026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.917015    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b6638d-2c7d-4360-b7f1-685edd3c0589" containerName="manager"
Sep 11 01:06:37.917066 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.917028    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b6638d-2c7d-4360-b7f1-685edd3c0589" containerName="manager"
Sep 11 01:06:37.917173 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.917097    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="11b6638d-2c7d-4360-b7f1-685edd3c0589" containerName="manager"
Sep 11 01:06:37.919010 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.918993    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:37.939751 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:37.939726    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"]
Sep 11 01:06:38.010904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.010875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rw55\" (UniqueName: \"kubernetes.io/projected/0cd59c8e-dff7-4344-9146-f1f2d1411d55-kube-api-access-4rw55\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bx6xn\" (UID: \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:38.011041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.010914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cd59c8e-dff7-4344-9146-f1f2d1411d55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bx6xn\" (UID: \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:38.011041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.010994    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11b6638d-2c7d-4360-b7f1-685edd3c0589-extensions-socket-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:06:38.011041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.011008    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92p4t\" (UniqueName: \"kubernetes.io/projected/11b6638d-2c7d-4360-b7f1-685edd3c0589-kube-api-access-92p4t\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:06:38.111916 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.111886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rw55\" (UniqueName: \"kubernetes.io/projected/0cd59c8e-dff7-4344-9146-f1f2d1411d55-kube-api-access-4rw55\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bx6xn\" (UID: \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:38.112110 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.111939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cd59c8e-dff7-4344-9146-f1f2d1411d55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bx6xn\" (UID: \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:38.112427 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.112405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cd59c8e-dff7-4344-9146-f1f2d1411d55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bx6xn\" (UID: \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:38.129171 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.129143    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rw55\" (UniqueName: \"kubernetes.io/projected/0cd59c8e-dff7-4344-9146-f1f2d1411d55-kube-api-access-4rw55\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bx6xn\" (UID: \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:38.228899 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.228859    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:38.503672 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.503592    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"]
Sep 11 01:06:38.507010 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:06:38.506982    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd59c8e_dff7_4344_9146_f1f2d1411d55.slice/crio-b33bf5b27d205f8b9778a1a092807f3ea9f57599e3921305fff34c31bcc6df08 WatchSource:0}: Error finding container b33bf5b27d205f8b9778a1a092807f3ea9f57599e3921305fff34c31bcc6df08: Status 404 returned error can't find the container with id b33bf5b27d205f8b9778a1a092807f3ea9f57599e3921305fff34c31bcc6df08
Sep 11 01:06:38.540419 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.540395    2583 generic.go:356] "Generic (PLEG): container finished" podID="11b6638d-2c7d-4360-b7f1-685edd3c0589" containerID="4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f" exitCode=0
Sep 11 01:06:38.540508 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.540439    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"
Sep 11 01:06:38.540508 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.540474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg" event={"ID":"11b6638d-2c7d-4360-b7f1-685edd3c0589","Type":"ContainerDied","Data":"4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f"}
Sep 11 01:06:38.540594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.540518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg" event={"ID":"11b6638d-2c7d-4360-b7f1-685edd3c0589","Type":"ContainerDied","Data":"6f5fc90b8246d3bda26eac0e0211554eecaff8493507d45493f0d24f63daaf21"}
Sep 11 01:06:38.540594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.540540    2583 scope.go:117] "RemoveContainer" containerID="4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f"
Sep 11 01:06:38.541991 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.541970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn" event={"ID":"0cd59c8e-dff7-4344-9146-f1f2d1411d55","Type":"ContainerStarted","Data":"b33bf5b27d205f8b9778a1a092807f3ea9f57599e3921305fff34c31bcc6df08"}
Sep 11 01:06:38.549026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.549011    2583 scope.go:117] "RemoveContainer" containerID="4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f"
Sep 11 01:06:38.549395 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:38.549357    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f\": container with ID starting with 4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f not found: ID does not exist" containerID="4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f"
Sep 11 01:06:38.549489 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.549401    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f"} err="failed to get container status \"4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f\": rpc error: code = NotFound desc = could not find container \"4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f\": container with ID starting with 4bdcbf2d40bed7e41e0629c35b282b50e8c305934f96e99b856b884d3c5de25f not found: ID does not exist"
Sep 11 01:06:38.573110 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.573088    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"]
Sep 11 01:06:38.585077 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:38.585055    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nmbqg"]
Sep 11 01:06:39.035927 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:39.035892    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11b6638d-2c7d-4360-b7f1-685edd3c0589" path="/var/lib/kubelet/pods/11b6638d-2c7d-4360-b7f1-685edd3c0589/volumes"
Sep 11 01:06:39.547662 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:39.547622    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn" event={"ID":"0cd59c8e-dff7-4344-9146-f1f2d1411d55","Type":"ContainerStarted","Data":"2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4"}
Sep 11 01:06:39.548104 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:39.547698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:39.578384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:39.578340    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn" podStartSLOduration=2.5783008609999998 podStartE2EDuration="2.578300861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:39.57590135 +0000 UTC m=+529.112552550" watchObservedRunningTime="2026-09-11 01:06:39.578300861 +0000 UTC m=+529.114952061"
Sep 11 01:06:50.555823 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:50.555745    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:06:50.620606 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:50.620574    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"]
Sep 11 01:06:50.620841 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:50.620813    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs" podUID="0b515ad9-4d62-473c-ac35-a7364ef967e0" containerName="manager" containerID="cri-o://a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737" gracePeriod=10
Sep 11 01:06:50.870777 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:50.870756    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:51.011058 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.011032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b515ad9-4d62-473c-ac35-a7364ef967e0-extensions-socket-volume\") pod \"0b515ad9-4d62-473c-ac35-a7364ef967e0\" (UID: \"0b515ad9-4d62-473c-ac35-a7364ef967e0\") "
Sep 11 01:06:51.011208 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.011136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6ns2\" (UniqueName: \"kubernetes.io/projected/0b515ad9-4d62-473c-ac35-a7364ef967e0-kube-api-access-s6ns2\") pod \"0b515ad9-4d62-473c-ac35-a7364ef967e0\" (UID: \"0b515ad9-4d62-473c-ac35-a7364ef967e0\") "
Sep 11 01:06:51.011512 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.011481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b515ad9-4d62-473c-ac35-a7364ef967e0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0b515ad9-4d62-473c-ac35-a7364ef967e0" (UID: "0b515ad9-4d62-473c-ac35-a7364ef967e0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:06:51.013655 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.013621    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b515ad9-4d62-473c-ac35-a7364ef967e0-kube-api-access-s6ns2" (OuterVolumeSpecName: "kube-api-access-s6ns2") pod "0b515ad9-4d62-473c-ac35-a7364ef967e0" (UID: "0b515ad9-4d62-473c-ac35-a7364ef967e0"). InnerVolumeSpecName "kube-api-access-s6ns2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:06:51.112731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.112658    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ns2\" (UniqueName: \"kubernetes.io/projected/0b515ad9-4d62-473c-ac35-a7364ef967e0-kube-api-access-s6ns2\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:06:51.112731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.112692    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b515ad9-4d62-473c-ac35-a7364ef967e0-extensions-socket-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:06:51.604208 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.604169    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b515ad9-4d62-473c-ac35-a7364ef967e0" containerID="a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737" exitCode=0
Sep 11 01:06:51.604584 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.604214    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"
Sep 11 01:06:51.604584 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.604238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs" event={"ID":"0b515ad9-4d62-473c-ac35-a7364ef967e0","Type":"ContainerDied","Data":"a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737"}
Sep 11 01:06:51.604584 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.604288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs" event={"ID":"0b515ad9-4d62-473c-ac35-a7364ef967e0","Type":"ContainerDied","Data":"d513c111f8efb9c729303e798208970e2c524fee58a61813440adfd06acf149e"}
Sep 11 01:06:51.604584 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.604341    2583 scope.go:117] "RemoveContainer" containerID="a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737"
Sep 11 01:06:51.612325 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.612288    2583 scope.go:117] "RemoveContainer" containerID="a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737"
Sep 11 01:06:51.612589 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:06:51.612570    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737\": container with ID starting with a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737 not found: ID does not exist" containerID="a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737"
Sep 11 01:06:51.612635 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.612598    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737"} err="failed to get container status \"a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737\": rpc error: code = NotFound desc = could not find container \"a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737\": container with ID starting with a8e4575068405ed6950b6c6b78c7fc34c4a90073cb4d8269d5c59ddb6cb2c737 not found: ID does not exist"
Sep 11 01:06:51.620598 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.620579    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"]
Sep 11 01:06:51.626077 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:51.626053    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-5vhjs"]
Sep 11 01:06:53.036066 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:06:53.036033    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b515ad9-4d62-473c-ac35-a7364ef967e0" path="/var/lib/kubelet/pods/0b515ad9-4d62-473c-ac35-a7364ef967e0/volumes"
Sep 11 01:07:09.355983 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.355943    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:07:09.356465 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.356250    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b515ad9-4d62-473c-ac35-a7364ef967e0" containerName="manager"
Sep 11 01:07:09.356465 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.356262    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b515ad9-4d62-473c-ac35-a7364ef967e0" containerName="manager"
Sep 11 01:07:09.356465 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.356339    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b515ad9-4d62-473c-ac35-a7364ef967e0" containerName="manager"
Sep 11 01:07:09.358766 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.358748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:09.362084 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.362056    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 01:07:09.362212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.362112    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7d58x\""
Sep 11 01:07:09.373585 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.373566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:07:09.409250 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.409213    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:07:09.468093 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.468056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qwh\" (UniqueName: \"kubernetes.io/projected/be7c29fc-a1b6-415a-8e7a-cb28876bc726-kube-api-access-n6qwh\") pod \"limitador-limitador-84994f7ffc-qzt5d\" (UID: \"be7c29fc-a1b6-415a-8e7a-cb28876bc726\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:09.468244 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.468154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/be7c29fc-a1b6-415a-8e7a-cb28876bc726-config-file\") pod \"limitador-limitador-84994f7ffc-qzt5d\" (UID: \"be7c29fc-a1b6-415a-8e7a-cb28876bc726\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:09.568726 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.568693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/be7c29fc-a1b6-415a-8e7a-cb28876bc726-config-file\") pod \"limitador-limitador-84994f7ffc-qzt5d\" (UID: \"be7c29fc-a1b6-415a-8e7a-cb28876bc726\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:09.568881 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.568735    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qwh\" (UniqueName: \"kubernetes.io/projected/be7c29fc-a1b6-415a-8e7a-cb28876bc726-kube-api-access-n6qwh\") pod \"limitador-limitador-84994f7ffc-qzt5d\" (UID: \"be7c29fc-a1b6-415a-8e7a-cb28876bc726\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:09.569353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.569333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/be7c29fc-a1b6-415a-8e7a-cb28876bc726-config-file\") pod \"limitador-limitador-84994f7ffc-qzt5d\" (UID: \"be7c29fc-a1b6-415a-8e7a-cb28876bc726\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:09.578197 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.578170    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qwh\" (UniqueName: \"kubernetes.io/projected/be7c29fc-a1b6-415a-8e7a-cb28876bc726-kube-api-access-n6qwh\") pod \"limitador-limitador-84994f7ffc-qzt5d\" (UID: \"be7c29fc-a1b6-415a-8e7a-cb28876bc726\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:09.669093 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.669023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:09.700133 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.700105    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cbthd"]
Sep 11 01:07:09.702503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.702479    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd"
Sep 11 01:07:09.704593 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.704574    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rg9lb\""
Sep 11 01:07:09.711302 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.711275    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cbthd"]
Sep 11 01:07:09.871757 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.871724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdt8\" (UniqueName: \"kubernetes.io/projected/b068c426-0efc-4c16-bb3c-b896335c89f8-kube-api-access-nsdt8\") pod \"authorino-7bcfd8f9cd-cbthd\" (UID: \"b068c426-0efc-4c16-bb3c-b896335c89f8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd"
Sep 11 01:07:09.942299 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.942275    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:07:09.944521 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:07:09.944492    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7c29fc_a1b6_415a_8e7a_cb28876bc726.slice/crio-6bd06cf929d57875a0ef4a5f4d47792369da545b9dc7154420a2f7addb118a30 WatchSource:0}: Error finding container 6bd06cf929d57875a0ef4a5f4d47792369da545b9dc7154420a2f7addb118a30: Status 404 returned error can't find the container with id 6bd06cf929d57875a0ef4a5f4d47792369da545b9dc7154420a2f7addb118a30
Sep 11 01:07:09.972581 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.972560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdt8\" (UniqueName: \"kubernetes.io/projected/b068c426-0efc-4c16-bb3c-b896335c89f8-kube-api-access-nsdt8\") pod \"authorino-7bcfd8f9cd-cbthd\" (UID: \"b068c426-0efc-4c16-bb3c-b896335c89f8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd"
Sep 11 01:07:09.984240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:09.984218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdt8\" (UniqueName: \"kubernetes.io/projected/b068c426-0efc-4c16-bb3c-b896335c89f8-kube-api-access-nsdt8\") pod \"authorino-7bcfd8f9cd-cbthd\" (UID: \"b068c426-0efc-4c16-bb3c-b896335c89f8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd"
Sep 11 01:07:10.013738 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:10.013692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd"
Sep 11 01:07:10.282211 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:10.282139    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cbthd"]
Sep 11 01:07:10.286068 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:07:10.286038    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb068c426_0efc_4c16_bb3c_b896335c89f8.slice/crio-2664ebad65794bb041f0b0e90170e73c63d4c27bf457e1292122ee1951e7a4fb WatchSource:0}: Error finding container 2664ebad65794bb041f0b0e90170e73c63d4c27bf457e1292122ee1951e7a4fb: Status 404 returned error can't find the container with id 2664ebad65794bb041f0b0e90170e73c63d4c27bf457e1292122ee1951e7a4fb
Sep 11 01:07:10.687365 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:10.687334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd" event={"ID":"b068c426-0efc-4c16-bb3c-b896335c89f8","Type":"ContainerStarted","Data":"2664ebad65794bb041f0b0e90170e73c63d4c27bf457e1292122ee1951e7a4fb"}
Sep 11 01:07:10.688479 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:10.688446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d" event={"ID":"be7c29fc-a1b6-415a-8e7a-cb28876bc726","Type":"ContainerStarted","Data":"6bd06cf929d57875a0ef4a5f4d47792369da545b9dc7154420a2f7addb118a30"}
Sep 11 01:07:11.695121 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:11.695085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d" event={"ID":"be7c29fc-a1b6-415a-8e7a-cb28876bc726","Type":"ContainerStarted","Data":"af213e6f301922d1a4688f91ac3c3c0b7d415ce733a073ebdfa4bcc2c661626c"}
Sep 11 01:07:11.695562 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:11.695232    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:11.711110 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:11.711054    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d" podStartSLOduration=1.451774961 podStartE2EDuration="2.711035807s" totalImagesPullingTime="1.259260846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:07:11.709633384 +0000 UTC m=+561.246284596" watchObservedRunningTime="2026-09-11 01:07:11.711035807 +0000 UTC m=+561.247687026"
Sep 11 01:07:12.700573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:12.700541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd" event={"ID":"b068c426-0efc-4c16-bb3c-b896335c89f8","Type":"ContainerStarted","Data":"8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4"}
Sep 11 01:07:12.714602 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:12.714558    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd" podStartSLOduration=2.161251874 podStartE2EDuration="3.714542495s" totalImagesPullingTime="1.553290621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:07:12.713850353 +0000 UTC m=+562.250501568" watchObservedRunningTime="2026-09-11 01:07:12.714542495 +0000 UTC m=+562.251193763"
Sep 11 01:07:13.494704 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:13.494674    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cbthd"]
Sep 11 01:07:14.709445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:14.709408    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd" podUID="b068c426-0efc-4c16-bb3c-b896335c89f8" containerName="authorino" containerID="cri-o://8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4" gracePeriod=30
Sep 11 01:07:14.950752 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:14.950732    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd"
Sep 11 01:07:15.115290 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.115253    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsdt8\" (UniqueName: \"kubernetes.io/projected/b068c426-0efc-4c16-bb3c-b896335c89f8-kube-api-access-nsdt8\") pod \"b068c426-0efc-4c16-bb3c-b896335c89f8\" (UID: \"b068c426-0efc-4c16-bb3c-b896335c89f8\") "
Sep 11 01:07:15.117449 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.117418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b068c426-0efc-4c16-bb3c-b896335c89f8-kube-api-access-nsdt8" (OuterVolumeSpecName: "kube-api-access-nsdt8") pod "b068c426-0efc-4c16-bb3c-b896335c89f8" (UID: "b068c426-0efc-4c16-bb3c-b896335c89f8"). InnerVolumeSpecName "kube-api-access-nsdt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:07:15.216917 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.216883    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdt8\" (UniqueName: \"kubernetes.io/projected/b068c426-0efc-4c16-bb3c-b896335c89f8-kube-api-access-nsdt8\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:07:15.717029 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.716994    2583 generic.go:356] "Generic (PLEG): container finished" podID="b068c426-0efc-4c16-bb3c-b896335c89f8" containerID="8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4" exitCode=0
Sep 11 01:07:15.717470 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.717035    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd"
Sep 11 01:07:15.717470 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.717055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd" event={"ID":"b068c426-0efc-4c16-bb3c-b896335c89f8","Type":"ContainerDied","Data":"8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4"}
Sep 11 01:07:15.717470 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.717086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cbthd" event={"ID":"b068c426-0efc-4c16-bb3c-b896335c89f8","Type":"ContainerDied","Data":"2664ebad65794bb041f0b0e90170e73c63d4c27bf457e1292122ee1951e7a4fb"}
Sep 11 01:07:15.717470 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.717103    2583 scope.go:117] "RemoveContainer" containerID="8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4"
Sep 11 01:07:15.726230 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.726208    2583 scope.go:117] "RemoveContainer" containerID="8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4"
Sep 11 01:07:15.726501 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:07:15.726482    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4\": container with ID starting with 8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4 not found: ID does not exist" containerID="8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4"
Sep 11 01:07:15.726587 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.726509    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4"} err="failed to get container status \"8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4\": rpc error: code = NotFound desc = could not find container \"8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4\": container with ID starting with 8558599a812f370ac2c7ef284c0d9ef1ba59e4c15f6d3dc8357ccd05d32c74e4 not found: ID does not exist"
Sep 11 01:07:15.737188 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.737165    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cbthd"]
Sep 11 01:07:15.742552 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:15.742530    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cbthd"]
Sep 11 01:07:17.035744 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:17.035714    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b068c426-0efc-4c16-bb3c-b896335c89f8" path="/var/lib/kubelet/pods/b068c426-0efc-4c16-bb3c-b896335c89f8/volumes"
Sep 11 01:07:22.702484 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:22.702447    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"
Sep 11 01:07:40.656793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.656756    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-75x6d"]
Sep 11 01:07:40.657255 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.657082    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b068c426-0efc-4c16-bb3c-b896335c89f8" containerName="authorino"
Sep 11 01:07:40.657255 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.657092    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b068c426-0efc-4c16-bb3c-b896335c89f8" containerName="authorino"
Sep 11 01:07:40.657255 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.657150    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b068c426-0efc-4c16-bb3c-b896335c89f8" containerName="authorino"
Sep 11 01:07:40.659543 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.659517    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-75x6d"
Sep 11 01:07:40.661647 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.661625    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rg9lb\""
Sep 11 01:07:40.667018 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.666994    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-75x6d"]
Sep 11 01:07:40.728195 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.728164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766qw\" (UniqueName: \"kubernetes.io/projected/0c602578-3fe8-4d0d-92ca-0cda11cf3f86-kube-api-access-766qw\") pod \"authorino-557f876f69-75x6d\" (UID: \"0c602578-3fe8-4d0d-92ca-0cda11cf3f86\") " pod="kuadrant-system/authorino-557f876f69-75x6d"
Sep 11 01:07:40.730688 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.730662    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-75x6d"]
Sep 11 01:07:40.730911 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:07:40.730892    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-766qw], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-75x6d" podUID="0c602578-3fe8-4d0d-92ca-0cda11cf3f86"
Sep 11 01:07:40.761578 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.761556    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mv2vn"]
Sep 11 01:07:40.763518 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.763502    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:40.769438 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.769292    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 01:07:40.783662 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.783644    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mv2vn"]
Sep 11 01:07:40.826176 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.826146    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-75x6d"
Sep 11 01:07:40.828723 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.828698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwq6\" (UniqueName: \"kubernetes.io/projected/a540ad47-e129-47fe-8450-2e7f137d265d-kube-api-access-ltwq6\") pod \"authorino-6d4dfc649f-mv2vn\" (UID: \"a540ad47-e129-47fe-8450-2e7f137d265d\") " pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:40.828852 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.828761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a540ad47-e129-47fe-8450-2e7f137d265d-tls-cert\") pod \"authorino-6d4dfc649f-mv2vn\" (UID: \"a540ad47-e129-47fe-8450-2e7f137d265d\") " pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:40.828852 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.828809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766qw\" (UniqueName: \"kubernetes.io/projected/0c602578-3fe8-4d0d-92ca-0cda11cf3f86-kube-api-access-766qw\") pod \"authorino-557f876f69-75x6d\" (UID: \"0c602578-3fe8-4d0d-92ca-0cda11cf3f86\") " pod="kuadrant-system/authorino-557f876f69-75x6d"
Sep 11 01:07:40.831787 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.831770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-75x6d"
Sep 11 01:07:40.840474 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.840454    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-766qw\" (UniqueName: \"kubernetes.io/projected/0c602578-3fe8-4d0d-92ca-0cda11cf3f86-kube-api-access-766qw\") pod \"authorino-557f876f69-75x6d\" (UID: \"0c602578-3fe8-4d0d-92ca-0cda11cf3f86\") " pod="kuadrant-system/authorino-557f876f69-75x6d"
Sep 11 01:07:40.917926 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.917846    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mv2vn"]
Sep 11 01:07:40.918088 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:07:40.918070    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ltwq6 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-mv2vn" podUID="a540ad47-e129-47fe-8450-2e7f137d265d"
Sep 11 01:07:40.929151 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.929120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-766qw\" (UniqueName: \"kubernetes.io/projected/0c602578-3fe8-4d0d-92ca-0cda11cf3f86-kube-api-access-766qw\") pod \"0c602578-3fe8-4d0d-92ca-0cda11cf3f86\" (UID: \"0c602578-3fe8-4d0d-92ca-0cda11cf3f86\") "
Sep 11 01:07:40.929294 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.929275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a540ad47-e129-47fe-8450-2e7f137d265d-tls-cert\") pod \"authorino-6d4dfc649f-mv2vn\" (UID: \"a540ad47-e129-47fe-8450-2e7f137d265d\") " pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:40.929402 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.929383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwq6\" (UniqueName: \"kubernetes.io/projected/a540ad47-e129-47fe-8450-2e7f137d265d-kube-api-access-ltwq6\") pod \"authorino-6d4dfc649f-mv2vn\" (UID: \"a540ad47-e129-47fe-8450-2e7f137d265d\") " pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:40.931228 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.931202    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c602578-3fe8-4d0d-92ca-0cda11cf3f86-kube-api-access-766qw" (OuterVolumeSpecName: "kube-api-access-766qw") pod "0c602578-3fe8-4d0d-92ca-0cda11cf3f86" (UID: "0c602578-3fe8-4d0d-92ca-0cda11cf3f86"). InnerVolumeSpecName "kube-api-access-766qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:07:40.931601 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.931582    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a540ad47-e129-47fe-8450-2e7f137d265d-tls-cert\") pod \"authorino-6d4dfc649f-mv2vn\" (UID: \"a540ad47-e129-47fe-8450-2e7f137d265d\") " pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:40.939115 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.939095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwq6\" (UniqueName: \"kubernetes.io/projected/a540ad47-e129-47fe-8450-2e7f137d265d-kube-api-access-ltwq6\") pod \"authorino-6d4dfc649f-mv2vn\" (UID: \"a540ad47-e129-47fe-8450-2e7f137d265d\") " pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:40.945154 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.945125    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67b8c575b5-mbjzd"]
Sep 11 01:07:40.947501 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.947481    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:07:40.954952 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:40.954928    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67b8c575b5-mbjzd"]
Sep 11 01:07:41.029832 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.029802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5642b6af-67d7-4b88-9dba-ae753bf77324-tls-cert\") pod \"authorino-67b8c575b5-mbjzd\" (UID: \"5642b6af-67d7-4b88-9dba-ae753bf77324\") " pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:07:41.029999 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.029843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/5642b6af-67d7-4b88-9dba-ae753bf77324-kube-api-access-tn4q2\") pod \"authorino-67b8c575b5-mbjzd\" (UID: \"5642b6af-67d7-4b88-9dba-ae753bf77324\") " pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:07:41.029999 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.029924    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766qw\" (UniqueName: \"kubernetes.io/projected/0c602578-3fe8-4d0d-92ca-0cda11cf3f86-kube-api-access-766qw\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:07:41.131196 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.131167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5642b6af-67d7-4b88-9dba-ae753bf77324-tls-cert\") pod \"authorino-67b8c575b5-mbjzd\" (UID: \"5642b6af-67d7-4b88-9dba-ae753bf77324\") " pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:07:41.131374 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.131202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/5642b6af-67d7-4b88-9dba-ae753bf77324-kube-api-access-tn4q2\") pod \"authorino-67b8c575b5-mbjzd\" (UID: \"5642b6af-67d7-4b88-9dba-ae753bf77324\") " pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:07:41.133594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.133571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5642b6af-67d7-4b88-9dba-ae753bf77324-tls-cert\") pod \"authorino-67b8c575b5-mbjzd\" (UID: \"5642b6af-67d7-4b88-9dba-ae753bf77324\") " pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:07:41.139299 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.139277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/5642b6af-67d7-4b88-9dba-ae753bf77324-kube-api-access-tn4q2\") pod \"authorino-67b8c575b5-mbjzd\" (UID: \"5642b6af-67d7-4b88-9dba-ae753bf77324\") " pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:07:41.257281 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.257198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:07:41.524340 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.524286    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67b8c575b5-mbjzd"]
Sep 11 01:07:41.526272 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:07:41.526245    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5642b6af_67d7_4b88_9dba_ae753bf77324.slice/crio-48b3b1bc4e535c7111699a925d7914157cb7e3d8f840f083aac1f7b6a36c1c6f WatchSource:0}: Error finding container 48b3b1bc4e535c7111699a925d7914157cb7e3d8f840f083aac1f7b6a36c1c6f: Status 404 returned error can't find the container with id 48b3b1bc4e535c7111699a925d7914157cb7e3d8f840f083aac1f7b6a36c1c6f
Sep 11 01:07:41.831785 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.831755    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b8c575b5-mbjzd" event={"ID":"5642b6af-67d7-4b88-9dba-ae753bf77324","Type":"ContainerStarted","Data":"48b3b1bc4e535c7111699a925d7914157cb7e3d8f840f083aac1f7b6a36c1c6f"}
Sep 11 01:07:41.832163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.831797    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:41.832163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.831826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-75x6d"
Sep 11 01:07:41.841381 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.841360    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:41.866279 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.866251    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-75x6d"]
Sep 11 01:07:41.867611 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.867594    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-75x6d"]
Sep 11 01:07:41.937691 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.937666    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a540ad47-e129-47fe-8450-2e7f137d265d-tls-cert\") pod \"a540ad47-e129-47fe-8450-2e7f137d265d\" (UID: \"a540ad47-e129-47fe-8450-2e7f137d265d\") "
Sep 11 01:07:41.937805 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.937706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltwq6\" (UniqueName: \"kubernetes.io/projected/a540ad47-e129-47fe-8450-2e7f137d265d-kube-api-access-ltwq6\") pod \"a540ad47-e129-47fe-8450-2e7f137d265d\" (UID: \"a540ad47-e129-47fe-8450-2e7f137d265d\") "
Sep 11 01:07:41.939705 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.939676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a540ad47-e129-47fe-8450-2e7f137d265d-kube-api-access-ltwq6" (OuterVolumeSpecName: "kube-api-access-ltwq6") pod "a540ad47-e129-47fe-8450-2e7f137d265d" (UID: "a540ad47-e129-47fe-8450-2e7f137d265d"). InnerVolumeSpecName "kube-api-access-ltwq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:07:41.939705 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:41.939680    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a540ad47-e129-47fe-8450-2e7f137d265d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a540ad47-e129-47fe-8450-2e7f137d265d" (UID: "a540ad47-e129-47fe-8450-2e7f137d265d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:07:42.038156 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:42.038125    2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a540ad47-e129-47fe-8450-2e7f137d265d-tls-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:07:42.038156 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:42.038151    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwq6\" (UniqueName: \"kubernetes.io/projected/a540ad47-e129-47fe-8450-2e7f137d265d-kube-api-access-ltwq6\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:07:42.837825 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:42.837787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b8c575b5-mbjzd" event={"ID":"5642b6af-67d7-4b88-9dba-ae753bf77324","Type":"ContainerStarted","Data":"84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c"}
Sep 11 01:07:42.838278 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:42.837799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mv2vn"
Sep 11 01:07:42.853601 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:42.853557    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-67b8c575b5-mbjzd" podStartSLOduration=2.295328353 podStartE2EDuration="2.853544344s" totalImagesPullingTime="558.215991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:07:42.852459799 +0000 UTC m=+592.389110995" watchObservedRunningTime="2026-09-11 01:07:42.853544344 +0000 UTC m=+592.390195545"
Sep 11 01:07:42.875714 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:42.875686    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mv2vn"]
Sep 11 01:07:42.882294 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:42.882265    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mv2vn"]
Sep 11 01:07:43.042278 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:43.042244    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c602578-3fe8-4d0d-92ca-0cda11cf3f86" path="/var/lib/kubelet/pods/0c602578-3fe8-4d0d-92ca-0cda11cf3f86/volumes"
Sep 11 01:07:43.042542 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:43.042528    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a540ad47-e129-47fe-8450-2e7f137d265d" path="/var/lib/kubelet/pods/a540ad47-e129-47fe-8450-2e7f137d265d/volumes"
Sep 11 01:07:50.871511 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:50.871481    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:07:50.871918 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:50.871546    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:07:50.877151 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:50.877130    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:07:50.877261 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:07:50.877129    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:08:26.034045 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:08:26.033957    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:08:48.187700 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:08:48.187660    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:08:53.984748 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:08:53.984709    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:09:05.227915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:05.227871    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:09:18.386215 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:18.386168    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:09:40.833931 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.833895    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"]
Sep 11 01:09:40.837407 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.837386    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.842774 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.842756    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6tkxm\""
Sep 11 01:09:40.843769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.843515    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 01:09:40.843769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.843513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 01:09:40.843769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.843627    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 01:09:40.858595 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.858574    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"]
Sep 11 01:09:40.892482 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.892451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e29af54-d922-4450-9098-1369a671387b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.892620 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.892497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.892620 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.892561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrr9\" (UniqueName: \"kubernetes.io/projected/7e29af54-d922-4450-9098-1369a671387b-kube-api-access-tcrr9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.892620 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.892592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.892747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.892624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.892747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.892656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993030 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.992993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993030 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.993036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993280 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.993056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e29af54-d922-4450-9098-1369a671387b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993280 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.993091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993280 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.993148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrr9\" (UniqueName: \"kubernetes.io/projected/7e29af54-d922-4450-9098-1369a671387b-kube-api-access-tcrr9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993280 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.993189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993496 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.993468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993619 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.993514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.993619 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.993527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.995448 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.995424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e29af54-d922-4450-9098-1369a671387b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:40.995599 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:40.995583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e29af54-d922-4450-9098-1369a671387b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:41.007940 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:41.007914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrr9\" (UniqueName: \"kubernetes.io/projected/7e29af54-d922-4450-9098-1369a671387b-kube-api-access-tcrr9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7\" (UID: \"7e29af54-d922-4450-9098-1369a671387b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:41.146872 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:41.146795    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:41.428343 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:41.428248    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"]
Sep 11 01:09:41.431683 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:09:41.431655    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e29af54_d922_4450_9098_1369a671387b.slice/crio-4bc9f076430d53fedfe0bc4dff03d6c8a3db6f570d13415d4d9a46e79b2efda2 WatchSource:0}: Error finding container 4bc9f076430d53fedfe0bc4dff03d6c8a3db6f570d13415d4d9a46e79b2efda2: Status 404 returned error can't find the container with id 4bc9f076430d53fedfe0bc4dff03d6c8a3db6f570d13415d4d9a46e79b2efda2
Sep 11 01:09:41.433267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:41.433250    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 01:09:42.368199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:42.368157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7" event={"ID":"7e29af54-d922-4450-9098-1369a671387b","Type":"ContainerStarted","Data":"4bc9f076430d53fedfe0bc4dff03d6c8a3db6f570d13415d4d9a46e79b2efda2"}
Sep 11 01:09:42.693155 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:42.693080    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:09:43.375331 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:43.375223    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7" event={"ID":"7e29af54-d922-4450-9098-1369a671387b","Type":"ContainerStarted","Data":"a8873b58e8d13f443f22d88b012ca7375f3e90a6040d5c286eb38d86daeafd85"}
Sep 11 01:09:43.427590 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:43.427539    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7" podStartSLOduration=2.406566609 podStartE2EDuration="3.427523571s" totalImagesPullingTime="1.020956962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:09:43.424798167 +0000 UTC m=+712.961449362" watchObservedRunningTime="2026-09-11 01:09:43.427523571 +0000 UTC m=+712.964174829"
Sep 11 01:09:51.147221 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:51.147182    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:51.147667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:51.147265    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:51.160002 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:51.159975    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:51.422547 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:51.422472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7"
Sep 11 01:09:52.489955 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.489924    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"]
Sep 11 01:09:52.493364 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.493347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.495938 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.495916    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 01:09:52.509601 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.509575    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"]
Sep 11 01:09:52.586340 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.586280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888d9584-591d-43bc-b31d-50580a2f1b27-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.586498 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.586375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.586498 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.586421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.586498 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.586445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7bs\" (UniqueName: \"kubernetes.io/projected/888d9584-591d-43bc-b31d-50580a2f1b27-kube-api-access-vx7bs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.586498 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.586483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.586734 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.586548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.687583 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.687553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.687583 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.687589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.687782 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.687633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888d9584-591d-43bc-b31d-50580a2f1b27-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.687782 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.687661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.687782 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.687687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.687782 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.687718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7bs\" (UniqueName: \"kubernetes.io/projected/888d9584-591d-43bc-b31d-50580a2f1b27-kube-api-access-vx7bs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.688089 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.688064    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.688158 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.688086    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.688158 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.688127    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.689757 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.689739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888d9584-591d-43bc-b31d-50580a2f1b27-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.690005 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.689989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888d9584-591d-43bc-b31d-50580a2f1b27-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.698488 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.698467    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7bs\" (UniqueName: \"kubernetes.io/projected/888d9584-591d-43bc-b31d-50580a2f1b27-kube-api-access-vx7bs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vq56h\" (UID: \"888d9584-591d-43bc-b31d-50580a2f1b27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:52.803424 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:52.803340    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:09:53.076503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:53.076479    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"]
Sep 11 01:09:53.078320 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:09:53.078278    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888d9584_591d_43bc_b31d_50580a2f1b27.slice/crio-92b8b2c201083e271d070937064e5f77c3d462dd5dd6644899fc9c253c5f289b WatchSource:0}: Error finding container 92b8b2c201083e271d070937064e5f77c3d462dd5dd6644899fc9c253c5f289b: Status 404 returned error can't find the container with id 92b8b2c201083e271d070937064e5f77c3d462dd5dd6644899fc9c253c5f289b
Sep 11 01:09:53.422459 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:53.422372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h" event={"ID":"888d9584-591d-43bc-b31d-50580a2f1b27","Type":"ContainerStarted","Data":"362a90d1df931248f7e8e63f95e627a8314933168c6ebe2e81acf0580236db0b"}
Sep 11 01:09:53.422459 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:53.422427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h" event={"ID":"888d9584-591d-43bc-b31d-50580a2f1b27","Type":"ContainerStarted","Data":"92b8b2c201083e271d070937064e5f77c3d462dd5dd6644899fc9c253c5f289b"}
Sep 11 01:09:53.442138 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:53.442088    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h" podStartSLOduration=1.264782542 podStartE2EDuration="1.442071736s" totalImagesPullingTime="177.289194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:09:53.440943857 +0000 UTC m=+722.977595059" watchObservedRunningTime="2026-09-11 01:09:53.442071736 +0000 UTC m=+722.978722936"
Sep 11 01:09:54.525383 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:09:54.525334    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:10:02.586985 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.586949    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"]
Sep 11 01:10:02.590374 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.590353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.592841 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.592819    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 01:10:02.600154 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.600135    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"]
Sep 11 01:10:02.671723 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.671695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.671872 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.671733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcg5\" (UniqueName: \"kubernetes.io/projected/037fd866-ebac-4e5a-a139-8790d0282f6c-kube-api-access-bpcg5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.671872 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.671769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.671872 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.671790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.671872 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.671810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037fd866-ebac-4e5a-a139-8790d0282f6c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.671872 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.671836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.772833 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.772801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.772992 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.772843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcg5\" (UniqueName: \"kubernetes.io/projected/037fd866-ebac-4e5a-a139-8790d0282f6c-kube-api-access-bpcg5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.772992 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.772878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.772992 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.772906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.772992 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.772931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037fd866-ebac-4e5a-a139-8790d0282f6c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.773208 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.773096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.773349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.773301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.773511 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.773481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.773618 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.773520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.775728 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.775707    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037fd866-ebac-4e5a-a139-8790d0282f6c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.775977 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.775963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037fd866-ebac-4e5a-a139-8790d0282f6c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.780849 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.780827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcg5\" (UniqueName: \"kubernetes.io/projected/037fd866-ebac-4e5a-a139-8790d0282f6c-kube-api-access-bpcg5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp\" (UID: \"037fd866-ebac-4e5a-a139-8790d0282f6c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:02.803513 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.803489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:10:02.803613 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.803564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:10:02.816148 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.816131    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:10:02.900037 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:02.899960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:03.176586 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:03.176542    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"]
Sep 11 01:10:03.177998 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:10:03.177974    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod037fd866_ebac_4e5a_a139_8790d0282f6c.slice/crio-8cc4923dc0239d02fbe9f8827a0255d5af51ea34721b5c79b2993edf363abd62 WatchSource:0}: Error finding container 8cc4923dc0239d02fbe9f8827a0255d5af51ea34721b5c79b2993edf363abd62: Status 404 returned error can't find the container with id 8cc4923dc0239d02fbe9f8827a0255d5af51ea34721b5c79b2993edf363abd62
Sep 11 01:10:03.469170 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:03.469080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp" event={"ID":"037fd866-ebac-4e5a-a139-8790d0282f6c","Type":"ContainerStarted","Data":"fdace260bb24e17c59ead72ab518712fb40870cc6631220f72dd2878c9502b58"}
Sep 11 01:10:03.469170 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:03.469127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp" event={"ID":"037fd866-ebac-4e5a-a139-8790d0282f6c","Type":"ContainerStarted","Data":"8cc4923dc0239d02fbe9f8827a0255d5af51ea34721b5c79b2993edf363abd62"}
Sep 11 01:10:03.481050 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:03.481026    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vq56h"
Sep 11 01:10:03.496424 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:03.496372    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp" podStartSLOduration=1.30379182 podStartE2EDuration="1.496357771s" totalImagesPullingTime="192.565951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:03.494277158 +0000 UTC m=+733.030928358" watchObservedRunningTime="2026-09-11 01:10:03.496357771 +0000 UTC m=+733.033008972"
Sep 11 01:10:05.090055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:05.090018    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:10:12.900280 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:12.900233    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:12.900818 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:12.900522    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:12.912915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:12.912890    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:10:13.526356 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:10:13.526328    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp"
Sep 11 01:11:57.720558 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:57.720520    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67b8c575b5-mbjzd"]
Sep 11 01:11:57.721446 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:57.721411    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-67b8c575b5-mbjzd" podUID="5642b6af-67d7-4b88-9dba-ae753bf77324" containerName="authorino" containerID="cri-o://84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c" gracePeriod=30
Sep 11 01:11:57.968813 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:57.968787    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:11:57.999898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:57.999793    2583 generic.go:356] "Generic (PLEG): container finished" podID="5642b6af-67d7-4b88-9dba-ae753bf77324" containerID="84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c" exitCode=0
Sep 11 01:11:57.999898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:57.999851    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b8c575b5-mbjzd"
Sep 11 01:11:57.999898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:57.999875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b8c575b5-mbjzd" event={"ID":"5642b6af-67d7-4b88-9dba-ae753bf77324","Type":"ContainerDied","Data":"84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c"}
Sep 11 01:11:58.000153 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:57.999924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b8c575b5-mbjzd" event={"ID":"5642b6af-67d7-4b88-9dba-ae753bf77324","Type":"ContainerDied","Data":"48b3b1bc4e535c7111699a925d7914157cb7e3d8f840f083aac1f7b6a36c1c6f"}
Sep 11 01:11:58.000153 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:57.999945    2583 scope.go:117] "RemoveContainer" containerID="84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c"
Sep 11 01:11:58.008761 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.008742    2583 scope.go:117] "RemoveContainer" containerID="84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c"
Sep 11 01:11:58.009030 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:11:58.009012    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c\": container with ID starting with 84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c not found: ID does not exist" containerID="84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c"
Sep 11 01:11:58.009116 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.009038    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c"} err="failed to get container status \"84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c\": rpc error: code = NotFound desc = could not find container \"84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c\": container with ID starting with 84aea5e77eca1ff6858ffa7b6ed15f5f33da295475710ee84c0b1da14a5d086c not found: ID does not exist"
Sep 11 01:11:58.057582 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.057556    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5642b6af-67d7-4b88-9dba-ae753bf77324-tls-cert\") pod \"5642b6af-67d7-4b88-9dba-ae753bf77324\" (UID: \"5642b6af-67d7-4b88-9dba-ae753bf77324\") "
Sep 11 01:11:58.057737 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.057616    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/5642b6af-67d7-4b88-9dba-ae753bf77324-kube-api-access-tn4q2\") pod \"5642b6af-67d7-4b88-9dba-ae753bf77324\" (UID: \"5642b6af-67d7-4b88-9dba-ae753bf77324\") "
Sep 11 01:11:58.059610 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.059569    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5642b6af-67d7-4b88-9dba-ae753bf77324-kube-api-access-tn4q2" (OuterVolumeSpecName: "kube-api-access-tn4q2") pod "5642b6af-67d7-4b88-9dba-ae753bf77324" (UID: "5642b6af-67d7-4b88-9dba-ae753bf77324"). InnerVolumeSpecName "kube-api-access-tn4q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:11:58.067198 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.067173    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5642b6af-67d7-4b88-9dba-ae753bf77324-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5642b6af-67d7-4b88-9dba-ae753bf77324" (UID: "5642b6af-67d7-4b88-9dba-ae753bf77324"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:11:58.158559 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.158523    2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5642b6af-67d7-4b88-9dba-ae753bf77324-tls-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:11:58.158559 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.158554    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4q2\" (UniqueName: \"kubernetes.io/projected/5642b6af-67d7-4b88-9dba-ae753bf77324-kube-api-access-tn4q2\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:11:58.340922 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.340894    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67b8c575b5-mbjzd"]
Sep 11 01:11:58.350469 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:58.350443    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-67b8c575b5-mbjzd"]
Sep 11 01:11:59.036633 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:11:59.036588    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5642b6af-67d7-4b88-9dba-ae753bf77324" path="/var/lib/kubelet/pods/5642b6af-67d7-4b88-9dba-ae753bf77324/volumes"
Sep 11 01:12:19.941175 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:19.941146    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:12:22.226188 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.226155    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"]
Sep 11 01:12:22.226833 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.226802    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5642b6af-67d7-4b88-9dba-ae753bf77324" containerName="authorino"
Sep 11 01:12:22.226833 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.226824    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5642b6af-67d7-4b88-9dba-ae753bf77324" containerName="authorino"
Sep 11 01:12:22.226979 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.226905    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5642b6af-67d7-4b88-9dba-ae753bf77324" containerName="authorino"
Sep 11 01:12:22.229862 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.229841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.232276 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.232255    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:12:22.233186 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.233159    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f4de10-serving-cert\""
Sep 11 01:12:22.233333 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.233237    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 01:12:22.233333 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.233268    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 01:12:22.233333 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.233219    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fdvhg\""
Sep 11 01:12:22.245241 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.245217    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"]
Sep 11 01:12:22.271712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.271683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e3f2ae-3aa4-4b66-8142-e82e167b2148-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.271843 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.271728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-metrics-certs\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.271921 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.271842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.271921 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.271877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4dm\" (UniqueName: \"kubernetes.io/projected/19e3f2ae-3aa4-4b66-8142-e82e167b2148-kube-api-access-wb4dm\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.372916 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.372868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.372916 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.372923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4dm\" (UniqueName: \"kubernetes.io/projected/19e3f2ae-3aa4-4b66-8142-e82e167b2148-kube-api-access-wb4dm\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.373163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.372963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e3f2ae-3aa4-4b66-8142-e82e167b2148-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.373163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.372987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-metrics-certs\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.373163 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:12:22.373021    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-serving-cert: secret "maas-api-e2e-worker-w0-f4de10-serving-cert" not found
Sep 11 01:12:22.373163 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:12:22.373091    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls podName:19e3f2ae-3aa4-4b66-8142-e82e167b2148 nodeName:}" failed. No retries permitted until 2026-09-11 01:12:22.873074369 +0000 UTC m=+872.409725547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls") pod "maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k" (UID: "19e3f2ae-3aa4-4b66-8142-e82e167b2148") : secret "maas-api-e2e-worker-w0-f4de10-serving-cert" not found
Sep 11 01:12:22.373751 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.373734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e3f2ae-3aa4-4b66-8142-e82e167b2148-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.375489 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.375466    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-metrics-certs\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.395721 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.395693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4dm\" (UniqueName: \"kubernetes.io/projected/19e3f2ae-3aa4-4b66-8142-e82e167b2148-kube-api-access-wb4dm\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.877719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:22.877684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:22.877927 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:12:22.877853    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-serving-cert: secret "maas-api-e2e-worker-w0-f4de10-serving-cert" not found
Sep 11 01:12:22.877998 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:12:22.877943    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls podName:19e3f2ae-3aa4-4b66-8142-e82e167b2148 nodeName:}" failed. No retries permitted until 2026-09-11 01:12:23.877921418 +0000 UTC m=+873.414572601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls") pod "maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k" (UID: "19e3f2ae-3aa4-4b66-8142-e82e167b2148") : secret "maas-api-e2e-worker-w0-f4de10-serving-cert" not found
Sep 11 01:12:23.885866 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:23.885829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:23.888175 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:23.888151    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:24.040169 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.040138    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:24.227261 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.227181    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"]
Sep 11 01:12:24.231224 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.231202    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:24.233450 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.233425    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-cd176e-dockercfg-pkrtb\""
Sep 11 01:12:24.233558 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.233464    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-cd176e\""
Sep 11 01:12:24.241096 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.241075    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"]
Sep 11 01:12:24.289054 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.289028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5\" (UID: \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:24.289168 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.289094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77b7l\" (UniqueName: \"kubernetes.io/projected/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-kube-api-access-77b7l\") pod \"payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5\" (UID: \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:24.314363 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.314343    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"]
Sep 11 01:12:24.316265 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:12:24.316226    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e3f2ae_3aa4_4b66_8142_e82e167b2148.slice/crio-6bce2f5aceab02c1975b98c6374b6599edb5cd1976228a58fba4b7cad05c584f WatchSource:0}: Error finding container 6bce2f5aceab02c1975b98c6374b6599edb5cd1976228a58fba4b7cad05c584f: Status 404 returned error can't find the container with id 6bce2f5aceab02c1975b98c6374b6599edb5cd1976228a58fba4b7cad05c584f
Sep 11 01:12:24.389919 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.389887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77b7l\" (UniqueName: \"kubernetes.io/projected/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-kube-api-access-77b7l\") pod \"payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5\" (UID: \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:24.390041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.389944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5\" (UID: \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:24.390631 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.390614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5\" (UID: \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:24.397759 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.397734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77b7l\" (UniqueName: \"kubernetes.io/projected/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-kube-api-access-77b7l\") pod \"payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5\" (UID: \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:24.542408 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.542337    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:24.816350 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:24.816249    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"]
Sep 11 01:12:24.816823 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:12:24.816783    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30fe5b7_b2d1_495b_bbd3_81c1dacee2bb.slice/crio-38543e1575b4d0798eb82e6425b139d5eba2326ac53b82847776a80e7d39e54a WatchSource:0}: Error finding container 38543e1575b4d0798eb82e6425b139d5eba2326ac53b82847776a80e7d39e54a: Status 404 returned error can't find the container with id 38543e1575b4d0798eb82e6425b139d5eba2326ac53b82847776a80e7d39e54a
Sep 11 01:12:25.127023 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.126940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5" event={"ID":"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb","Type":"ContainerStarted","Data":"38543e1575b4d0798eb82e6425b139d5eba2326ac53b82847776a80e7d39e54a"}
Sep 11 01:12:25.129149 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.129116    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k" event={"ID":"19e3f2ae-3aa4-4b66-8142-e82e167b2148","Type":"ContainerStarted","Data":"6bce2f5aceab02c1975b98c6374b6599edb5cd1976228a58fba4b7cad05c584f"}
Sep 11 01:12:25.772640 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.772593    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"]
Sep 11 01:12:25.783787 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.783759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.787069 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.786041    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fd82e780-openshift-default-dockercfg-spr7t\""
Sep 11 01:12:25.787069 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.786276    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 01:12:25.788407 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.788347    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"]
Sep 11 01:12:25.800908 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.800876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92b8037-43a3-4c30-aa65-7d75625ec224-istiod-ca-cert\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.801032 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.800925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-envoy\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.801032 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.800952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-podinfo\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.801032 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.800999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-credential-socket\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.801032 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.801025    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-certs\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.801249 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.801052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-data\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.801249 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.801118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-token\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.801249 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.801148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p6g8\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-kube-api-access-5p6g8\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.801249 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.801182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-socket\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.901999 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.901960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92b8037-43a3-4c30-aa65-7d75625ec224-istiod-ca-cert\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902196 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-envoy\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902196 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-podinfo\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902196 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-credential-socket\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902196 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-certs\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902196 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-data\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902464 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-token\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902464 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p6g8\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-kube-api-access-5p6g8\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902464 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-socket\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902756 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-certs\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.902880 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.902828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92b8037-43a3-4c30-aa65-7d75625ec224-istiod-ca-cert\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.903049 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.903013    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-credential-socket\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.903237 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.903213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-socket\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.903496 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.903451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-data\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.905228 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.905167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-envoy\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.905635 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.905608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-podinfo\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.919124 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.919086    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-token\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:25.919654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:25.919607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p6g8\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-kube-api-access-5p6g8\") pod \"e2e-ait-fd82e780-openshift-default-69744654bd-d295j\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") " pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:26.099537 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:26.099505    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:28.044610 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:28.044586    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"]
Sep 11 01:12:28.046673 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:12:28.046644    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92b8037_43a3_4c30_aa65_7d75625ec224.slice/crio-dce41abf777fa540251bd35b4a6531f21177168e600702c6456e6e5a84c54cc2 WatchSource:0}: Error finding container dce41abf777fa540251bd35b4a6531f21177168e600702c6456e6e5a84c54cc2: Status 404 returned error can't find the container with id dce41abf777fa540251bd35b4a6531f21177168e600702c6456e6e5a84c54cc2
Sep 11 01:12:28.150511 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:28.150476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5" event={"ID":"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb","Type":"ContainerStarted","Data":"27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6"}
Sep 11 01:12:28.150672 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:28.150654    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:28.152343 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:28.152300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k" event={"ID":"19e3f2ae-3aa4-4b66-8142-e82e167b2148","Type":"ContainerStarted","Data":"39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310"}
Sep 11 01:12:28.152498 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:28.152404    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:28.153662 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:28.153636    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" event={"ID":"f92b8037-43a3-4c30-aa65-7d75625ec224","Type":"ContainerStarted","Data":"dce41abf777fa540251bd35b4a6531f21177168e600702c6456e6e5a84c54cc2"}
Sep 11 01:12:28.168049 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:28.168002    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5" podStartSLOduration=1.162799612 podStartE2EDuration="4.167988519s" totalImagesPullingTime="3.005188907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:28.167277427 +0000 UTC m=+877.703928628" watchObservedRunningTime="2026-09-11 01:12:28.167988519 +0000 UTC m=+877.704639720"
Sep 11 01:12:28.188328 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:28.188129    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k" podStartSLOduration=2.726137892 podStartE2EDuration="6.188111857s" totalImagesPullingTime="3.461973965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:28.185601415 +0000 UTC m=+877.722252617" watchObservedRunningTime="2026-09-11 01:12:28.188111857 +0000 UTC m=+877.724763059"
Sep 11 01:12:30.395548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.395512    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"]
Sep 11 01:12:30.402461 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.402442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.405090 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.405062    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-2c1e69-serving-cert\""
Sep 11 01:12:30.409128 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.409104    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"]
Sep 11 01:12:30.446096 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.446061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8347d4c-111e-48d5-aa31-11bc0f188ff9-openshift-service-ca\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.446364 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.446167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.446364 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.446210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wfx\" (UniqueName: \"kubernetes.io/projected/e8347d4c-111e-48d5-aa31-11bc0f188ff9-kube-api-access-h2wfx\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.446364 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.446239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-metrics-certs\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.547819 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.547776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8347d4c-111e-48d5-aa31-11bc0f188ff9-openshift-service-ca\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.548021 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.547880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.548021 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.547923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wfx\" (UniqueName: \"kubernetes.io/projected/e8347d4c-111e-48d5-aa31-11bc0f188ff9-kube-api-access-h2wfx\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.548021 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.547954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-metrics-certs\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.548021 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:12:30.548044    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-serving-cert: secret "maas-api-e2e-autores-2c1e69-serving-cert" not found
Sep 11 01:12:30.548408 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:12:30.548124    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls podName:e8347d4c-111e-48d5-aa31-11bc0f188ff9 nodeName:}" failed. No retries permitted until 2026-09-11 01:12:31.048103714 +0000 UTC m=+880.584754916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls") pod "maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb" (UID: "e8347d4c-111e-48d5-aa31-11bc0f188ff9") : secret "maas-api-e2e-autores-2c1e69-serving-cert" not found
Sep 11 01:12:30.548673 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.548644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8347d4c-111e-48d5-aa31-11bc0f188ff9-openshift-service-ca\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.551047 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.551017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-metrics-certs\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.559732 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.559709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wfx\" (UniqueName: \"kubernetes.io/projected/e8347d4c-111e-48d5-aa31-11bc0f188ff9-kube-api-access-h2wfx\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:30.814276 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.814242    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:12:30.814395 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.814344    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:12:30.814395 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:30.814386    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:12:31.052559 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:31.052529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:31.054970 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:31.054947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls\") pod \"maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:31.179485 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:31.179400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" event={"ID":"f92b8037-43a3-4c30-aa65-7d75625ec224","Type":"ContainerStarted","Data":"6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7"}
Sep 11 01:12:31.201768 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:31.201718    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" podStartSLOduration=3.43701697 podStartE2EDuration="6.201700833s" totalImagesPullingTime="2.764683863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:31.200207145 +0000 UTC m=+880.736858347" watchObservedRunningTime="2026-09-11 01:12:31.201700833 +0000 UTC m=+880.738352034"
Sep 11 01:12:31.316514 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:31.316484    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:31.592891 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:31.592868    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"]
Sep 11 01:12:31.594922 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:12:31.594897    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8347d4c_111e_48d5_aa31_11bc0f188ff9.slice/crio-c35f2c609dff522417775a3f3b9060e66d1f92f9e5f59f89951abf5d1e107a5c WatchSource:0}: Error finding container c35f2c609dff522417775a3f3b9060e66d1f92f9e5f59f89951abf5d1e107a5c: Status 404 returned error can't find the container with id c35f2c609dff522417775a3f3b9060e66d1f92f9e5f59f89951abf5d1e107a5c
Sep 11 01:12:32.099700 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:32.099666    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:32.104611 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:32.104586    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:32.186665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:32.186578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb" event={"ID":"e8347d4c-111e-48d5-aa31-11bc0f188ff9","Type":"ContainerStarted","Data":"6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c"}
Sep 11 01:12:32.186869 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:32.186852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:32.186935 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:32.186883    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:32.186935 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:32.186900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb" event={"ID":"e8347d4c-111e-48d5-aa31-11bc0f188ff9","Type":"ContainerStarted","Data":"c35f2c609dff522417775a3f3b9060e66d1f92f9e5f59f89951abf5d1e107a5c"}
Sep 11 01:12:32.187864 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:32.187844    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:32.203670 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:32.203629    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb" podStartSLOduration=1.897045158 podStartE2EDuration="2.203615645s" totalImagesPullingTime="306.570487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:32.201078511 +0000 UTC m=+881.737729712" watchObservedRunningTime="2026-09-11 01:12:32.203615645 +0000 UTC m=+881.740266845"
Sep 11 01:12:34.170116 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:34.170085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:12:38.196964 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:38.196928    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:12:38.582435 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:38.582397    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:12:39.167756 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:39.167725    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:12:48.229466 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:48.229416    2583 patch_prober.go:28] interesting pod/e2e-ait-fd82e780-openshift-default-69744654bd-d295j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 01:12:48.229832 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:48.229486    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 01:12:50.284873 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.284837    2583 generic.go:356] "Generic (PLEG): container finished" podID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerID="6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7" exitCode=0
Sep 11 01:12:50.284873 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.284879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" event={"ID":"f92b8037-43a3-4c30-aa65-7d75625ec224","Type":"ContainerDied","Data":"6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7"}
Sep 11 01:12:50.285371 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.285286    2583 scope.go:117] "RemoveContainer" containerID="6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7"
Sep 11 01:12:50.287299 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.287268    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:12:50.287385 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.287358    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:12:50.287424 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.287390    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:12:50.903916 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.903885    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:12:50.904352 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.904333    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:12:50.909373 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.909355    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:12:50.909727 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:50.909712    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:12:51.291252 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.291171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" event={"ID":"f92b8037-43a3-4c30-aa65-7d75625ec224","Type":"ContainerStarted","Data":"cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225"}
Sep 11 01:12:51.345695 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.345662    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"]
Sep 11 01:12:51.393956 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.393921    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"]
Sep 11 01:12:51.394108 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.394013    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:51.396520 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.396488    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-10a215\""
Sep 11 01:12:51.396520 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.396509    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-10a215-dockercfg-gk9kr\""
Sep 11 01:12:51.559697 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.559664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1870644-b475-4d91-8e24-de649b329680-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2\" (UID: \"b1870644-b475-4d91-8e24-de649b329680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:51.559868 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.559714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpk5\" (UniqueName: \"kubernetes.io/projected/b1870644-b475-4d91-8e24-de649b329680-kube-api-access-vkpk5\") pod \"payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2\" (UID: \"b1870644-b475-4d91-8e24-de649b329680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:51.661164 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.661130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1870644-b475-4d91-8e24-de649b329680-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2\" (UID: \"b1870644-b475-4d91-8e24-de649b329680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:51.661164 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.661173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpk5\" (UniqueName: \"kubernetes.io/projected/b1870644-b475-4d91-8e24-de649b329680-kube-api-access-vkpk5\") pod \"payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2\" (UID: \"b1870644-b475-4d91-8e24-de649b329680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:51.661936 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.661904    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1870644-b475-4d91-8e24-de649b329680-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2\" (UID: \"b1870644-b475-4d91-8e24-de649b329680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:51.668758 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.668736    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpk5\" (UniqueName: \"kubernetes.io/projected/b1870644-b475-4d91-8e24-de649b329680-kube-api-access-vkpk5\") pod \"payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2\" (UID: \"b1870644-b475-4d91-8e24-de649b329680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:51.704268 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:51.704242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:52.007372 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:52.007342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"]
Sep 11 01:12:52.010170 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:12:52.010141    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1870644_b475_4d91_8e24_de649b329680.slice/crio-2a379e4953569c23e87464b1698b10845d3bfb0d80e2ad49eec6e6e8d6f7a235 WatchSource:0}: Error finding container 2a379e4953569c23e87464b1698b10845d3bfb0d80e2ad49eec6e6e8d6f7a235: Status 404 returned error can't find the container with id 2a379e4953569c23e87464b1698b10845d3bfb0d80e2ad49eec6e6e8d6f7a235
Sep 11 01:12:52.100629 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:52.100597    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:52.100806 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:52.100778    2583 patch_prober.go:28] interesting pod/e2e-ait-fd82e780-openshift-default-69744654bd-d295j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 01:12:52.100856 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:52.100830    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 01:12:52.298625 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:52.298549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2" event={"ID":"b1870644-b475-4d91-8e24-de649b329680","Type":"ContainerStarted","Data":"430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc"}
Sep 11 01:12:52.299028 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:52.299010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2" event={"ID":"b1870644-b475-4d91-8e24-de649b329680","Type":"ContainerStarted","Data":"2a379e4953569c23e87464b1698b10845d3bfb0d80e2ad49eec6e6e8d6f7a235"}
Sep 11 01:12:52.299124 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:52.299115    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:12:52.317338 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:52.317212    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2" podStartSLOduration=1.317199799 podStartE2EDuration="1.317199799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:52.315166952 +0000 UTC m=+901.851818156" watchObservedRunningTime="2026-09-11 01:12:52.317199799 +0000 UTC m=+901.853850999"
Sep 11 01:12:53.099818 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:53.099785    2583 patch_prober.go:28] interesting pod/e2e-ait-fd82e780-openshift-default-69744654bd-d295j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 01:12:53.099983 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:53.099836    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 01:12:54.100059 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:54.100019    2583 patch_prober.go:28] interesting pod/e2e-ait-fd82e780-openshift-default-69744654bd-d295j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 01:12:54.100573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:54.100085    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 01:12:55.100826 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:55.100775    2583 patch_prober.go:28] interesting pod/e2e-ait-fd82e780-openshift-default-69744654bd-d295j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 01:12:55.101210 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:55.100841    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 01:12:56.100007 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:56.099968    2583 patch_prober.go:28] interesting pod/e2e-ait-fd82e780-openshift-default-69744654bd-d295j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 01:12:56.100190 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:56.100027    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 01:12:57.103209 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:57.103177    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:57.103603 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:57.103584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:12:57.104146 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:12:57.104126    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:13:00.277322 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:00.277279    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:13:04.309945 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:04.309913    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:13:05.498841 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:05.498805    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"]
Sep 11 01:13:05.499330 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:05.499062    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy" containerID="cri-o://cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225" gracePeriod=30
Sep 11 01:13:06.574234 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.574194    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"]
Sep 11 01:13:06.577355 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.577337    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.579621 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.579596    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0b6ae428-gw-openshift-default-dockercfg-pk2jk\""
Sep 11 01:13:06.589100 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.589076    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"]
Sep 11 01:13:06.599555 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-certs\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.599645 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599572    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65588002-1822-4ca3-b8cd-001ee6ff2a72-istiod-ca-cert\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.599645 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-socket\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.599714 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtjw\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-kube-api-access-wvtjw\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.599714 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-envoy\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.599781 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-credential-socket\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.599781 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-podinfo\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.599843 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-data\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.599900 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.599878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-token\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.700829 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-credential-socket\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.700829 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-podinfo\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-data\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-token\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-certs\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65588002-1822-4ca3-b8cd-001ee6ff2a72-istiod-ca-cert\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-socket\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtjw\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-kube-api-access-wvtjw\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.700990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-envoy\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701405 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.701248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-data\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701405 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.701273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-credential-socket\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701405 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.701351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-certs\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701578 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.701558    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-socket\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.701796 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.701778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65588002-1822-4ca3-b8cd-001ee6ff2a72-istiod-ca-cert\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.703778 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.703751    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-envoy\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.703904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.703887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-podinfo\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.707999 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.707975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-token\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.708403 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.708383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtjw\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-kube-api-access-wvtjw\") pod \"e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") " pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:06.890272 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:06.890190    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:07.168691 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:07.168663    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"]
Sep 11 01:13:07.170683 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:13:07.170655    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65588002_1822_4ca3_b8cd_001ee6ff2a72.slice/crio-8349f94e810e1aa4ca3b7623369e60d6dbf5d9568bc265519bc69178e36aefed WatchSource:0}: Error finding container 8349f94e810e1aa4ca3b7623369e60d6dbf5d9568bc265519bc69178e36aefed: Status 404 returned error can't find the container with id 8349f94e810e1aa4ca3b7623369e60d6dbf5d9568bc265519bc69178e36aefed
Sep 11 01:13:07.173278 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:07.173248    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:07.173363 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:07.173326    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:07.173402 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:07.173376    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:07.373784 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:07.373745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" event={"ID":"65588002-1822-4ca3-b8cd-001ee6ff2a72","Type":"ContainerStarted","Data":"99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50"}
Sep 11 01:13:07.373956 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:07.373796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" event={"ID":"65588002-1822-4ca3-b8cd-001ee6ff2a72","Type":"ContainerStarted","Data":"8349f94e810e1aa4ca3b7623369e60d6dbf5d9568bc265519bc69178e36aefed"}
Sep 11 01:13:07.391719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:07.391675    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podStartSLOduration=1.391646805 podStartE2EDuration="1.391646805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:13:07.390568522 +0000 UTC m=+916.927219713" watchObservedRunningTime="2026-09-11 01:13:07.391646805 +0000 UTC m=+916.928298005"
Sep 11 01:13:07.891104 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:07.891071    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:08.895828 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:08.895797    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:09.383389 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:09.383351    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:09.384225 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:09.384201    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:10.763200 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.763143    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:13:10.839870 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.839842    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-podinfo\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.839879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-credential-socket\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.839901    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-certs\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.839928    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-data\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.839947    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-envoy\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.839968    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92b8037-43a3-4c30-aa65-7d75625ec224-istiod-ca-cert\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.839988    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-token\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840055 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.840013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-socket\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840409 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.840094    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p6g8\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-kube-api-access-5p6g8\") pod \"f92b8037-43a3-4c30-aa65-7d75625ec224\" (UID: \"f92b8037-43a3-4c30-aa65-7d75625ec224\") "
Sep 11 01:13:10.840409 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.840363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:10.840409 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.840371    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-data" (OuterVolumeSpecName: "istio-data") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:10.840571 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.840438    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:10.840571 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.840450    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92b8037-43a3-4c30-aa65-7d75625ec224-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:13:10.840684 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.840588    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:10.842369 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.842341    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:13:10.842499 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.842454    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:10.842499 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.842456    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-token" (OuterVolumeSpecName: "istio-token") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:13:10.842602 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.842509    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-kube-api-access-5p6g8" (OuterVolumeSpecName: "kube-api-access-5p6g8") pod "f92b8037-43a3-4c30-aa65-7d75625ec224" (UID: "f92b8037-43a3-4c30-aa65-7d75625ec224"). InnerVolumeSpecName "kube-api-access-5p6g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:13:10.940807 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940774    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:10.940807 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940802    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:10.940807 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940812    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92b8037-43a3-4c30-aa65-7d75625ec224-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:10.941027 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940820    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:10.941027 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940829    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:10.941027 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940837    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p6g8\" (UniqueName: \"kubernetes.io/projected/f92b8037-43a3-4c30-aa65-7d75625ec224-kube-api-access-5p6g8\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:10.941027 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940847    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92b8037-43a3-4c30-aa65-7d75625ec224-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:10.941027 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940855    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:10.941027 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:10.940864    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92b8037-43a3-4c30-aa65-7d75625ec224-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:11.395034 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.394997    2583 generic.go:356] "Generic (PLEG): container finished" podID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerID="cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225" exitCode=0
Sep 11 01:13:11.395199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.395049    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"
Sep 11 01:13:11.395199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.395077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" event={"ID":"f92b8037-43a3-4c30-aa65-7d75625ec224","Type":"ContainerDied","Data":"cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225"}
Sep 11 01:13:11.395199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.395141    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j" event={"ID":"f92b8037-43a3-4c30-aa65-7d75625ec224","Type":"ContainerDied","Data":"dce41abf777fa540251bd35b4a6531f21177168e600702c6456e6e5a84c54cc2"}
Sep 11 01:13:11.395199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.395170    2583 scope.go:117] "RemoveContainer" containerID="cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225"
Sep 11 01:13:11.404197 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.404183    2583 scope.go:117] "RemoveContainer" containerID="6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7"
Sep 11 01:13:11.412205 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.412178    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"]
Sep 11 01:13:11.412911 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.412894    2583 scope.go:117] "RemoveContainer" containerID="cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225"
Sep 11 01:13:11.413224 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:13:11.413198    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225\": container with ID starting with cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225 not found: ID does not exist" containerID="cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225"
Sep 11 01:13:11.413302 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.413230    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225"} err="failed to get container status \"cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225\": rpc error: code = NotFound desc = could not find container \"cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225\": container with ID starting with cb05c5616495bbf79bb79c2c9e1b7c008da1699fcacb0e6c8fe222f1c4099225 not found: ID does not exist"
Sep 11 01:13:11.413302 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.413248    2583 scope.go:117] "RemoveContainer" containerID="6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7"
Sep 11 01:13:11.413572 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:13:11.413551    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7\": container with ID starting with 6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7 not found: ID does not exist" containerID="6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7"
Sep 11 01:13:11.413613 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.413581    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7"} err="failed to get container status \"6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7\": rpc error: code = NotFound desc = could not find container \"6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7\": container with ID starting with 6532539fb3285dda4cfcb526f5e6484eba06a54fd6716836edccf9d1e32353e7 not found: ID does not exist"
Sep 11 01:13:11.422594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:11.422559    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fd82e780-openshift-default-69744654bd-d295j"]
Sep 11 01:13:13.037435 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:13.037400    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" path="/var/lib/kubelet/pods/f92b8037-43a3-4c30-aa65-7d75625ec224/volumes"
Sep 11 01:13:13.777517 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:13.777484    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:13:23.462796 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.462767    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"]
Sep 11 01:13:23.463147 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.463133    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy"
Sep 11 01:13:23.463147 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.463147    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy"
Sep 11 01:13:23.463219 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.463210    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy"
Sep 11 01:13:23.463252 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.463221    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy"
Sep 11 01:13:23.465135 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.465119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.467390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.467365    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f4de10\"/\"openshift-service-ca.crt\""
Sep 11 01:13:23.467496 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.467373    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f4de10\"/\"kube-root-ca.crt\""
Sep 11 01:13:23.468186 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.468169    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f4de10\"/\"default-dockercfg-mg9vz\""
Sep 11 01:13:23.468289 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.468185    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f4de10\"/\"simulator-w0-f4de10-kserve-self-signed-certs\""
Sep 11 01:13:23.477455 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.476963    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"]
Sep 11 01:13:23.559704 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.559676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5d6\" (UniqueName: \"kubernetes.io/projected/cc6db59b-0137-4c32-9a48-4a03c79a93d9-kube-api-access-4h5d6\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.559836 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.559714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-home\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.559836 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.559752    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-dshm\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.559836 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.559798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-model-cache\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.559959 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.559838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tls-certs\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.559959 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.559860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tmp-dir\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660370 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tls-certs\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660370 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tmp-dir\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660370 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5d6\" (UniqueName: \"kubernetes.io/projected/cc6db59b-0137-4c32-9a48-4a03c79a93d9-kube-api-access-4h5d6\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660659 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-home\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660659 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-dshm\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660659 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-model-cache\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660820 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tmp-dir\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660929 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-model-cache\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.660992 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.660984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-home\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.662683 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.662660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-dshm\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.662785 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.662757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tls-certs\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.667946 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.667927    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5d6\" (UniqueName: \"kubernetes.io/projected/cc6db59b-0137-4c32-9a48-4a03c79a93d9-kube-api-access-4h5d6\") pod \"simulator-w0-f4de10-kserve-5b9459b4b9-bckk8\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") " pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:23.838408 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:23.838368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:24.112645 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.112556    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"]
Sep 11 01:13:24.115402 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:13:24.115363    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc6db59b_0137_4c32_9a48_4a03c79a93d9.slice/crio-6be06eda2fbc88dad65ae23babc338160a938c8b8ac5c0ef451eea1a3abc2848 WatchSource:0}: Error finding container 6be06eda2fbc88dad65ae23babc338160a938c8b8ac5c0ef451eea1a3abc2848: Status 404 returned error can't find the container with id 6be06eda2fbc88dad65ae23babc338160a938c8b8ac5c0ef451eea1a3abc2848
Sep 11 01:13:24.384337 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.384244    2583 patch_prober.go:28] interesting pod/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 11 01:13:24.384337 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.384295    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 11 01:13:24.483792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.483760    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8" event={"ID":"cc6db59b-0137-4c32-9a48-4a03c79a93d9","Type":"ContainerStarted","Data":"569b04f3c44cdcbdd46359e326fe0c68f7e916b21cec49f1b681f6d7cec82167"}
Sep 11 01:13:24.484220 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.483803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8" event={"ID":"cc6db59b-0137-4c32-9a48-4a03c79a93d9","Type":"ContainerStarted","Data":"6be06eda2fbc88dad65ae23babc338160a938c8b8ac5c0ef451eea1a3abc2848"}
Sep 11 01:13:24.492300 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.492269    2583 generic.go:356] "Generic (PLEG): container finished" podID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerID="99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50" exitCode=0
Sep 11 01:13:24.492455 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.492330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" event={"ID":"65588002-1822-4ca3-b8cd-001ee6ff2a72","Type":"ContainerDied","Data":"99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50"}
Sep 11 01:13:24.492690 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.492670    2583 scope.go:117] "RemoveContainer" containerID="99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50"
Sep 11 01:13:24.493809 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.493779    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:24.493897 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.493866    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:24.493966 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.493913    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:24.502455 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:24.502415    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8" podStartSLOduration=1.5024037350000001 podStartE2EDuration="1.502403735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:13:24.500108156 +0000 UTC m=+934.036759356" watchObservedRunningTime="2026-09-11 01:13:24.502403735 +0000 UTC m=+934.039054935"
Sep 11 01:13:25.499213 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:25.499177    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" event={"ID":"65588002-1822-4ca3-b8cd-001ee6ff2a72","Type":"ContainerStarted","Data":"e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e"}
Sep 11 01:13:25.891217 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:25.891179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:25.891398 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:25.891376    2583 patch_prober.go:28] interesting pod/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 11 01:13:25.891463 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:25.891431    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 11 01:13:26.890523 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:26.890482    2583 patch_prober.go:28] interesting pod/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 11 01:13:26.890910 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:26.890541    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 11 01:13:27.891029 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:27.890997    2583 patch_prober.go:28] interesting pod/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 11 01:13:27.891413 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:27.891079    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 11 01:13:28.890880 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:28.890845    2583 patch_prober.go:28] interesting pod/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 11 01:13:28.891062 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:28.890903    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 11 01:13:29.890833 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:29.890796    2583 patch_prober.go:28] interesting pod/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 11 01:13:29.891009 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:29.890855    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 11 01:13:30.895146 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:30.895116    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:30.895573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:30.895404    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:30.896059 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:30.896043    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:32.990775 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:32.990739    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:13:33.839579 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:33.839542    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:33.839751 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:33.839597    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:33.852359 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:33.852331    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:34.557683 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:34.557656    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:13:40.844623 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:40.844555    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"]
Sep 11 01:13:40.845204 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:40.844777    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy" containerID="cri-o://e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e" gracePeriod=30
Sep 11 01:13:41.665680 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.665647    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"]
Sep 11 01:13:41.666106 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.666088    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy"
Sep 11 01:13:41.666193 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.666108    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92b8037-43a3-4c30-aa65-7d75625ec224" containerName="istio-proxy"
Sep 11 01:13:41.669092 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.669072    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.671503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.671484    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9516b91d-openshift-default-dockercfg-kj8v9\""
Sep 11 01:13:41.680417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.680389    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"]
Sep 11 01:13:41.830324 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-token\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.830324 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1085211a-36ba-4063-a3a3-51db5f20c16b-istiod-ca-cert\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.830518 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-credential-socket\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.830518 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-envoy\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.830518 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-data\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.830620 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-certs\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.830620 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nll\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-kube-api-access-98nll\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.830620 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-socket\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.830714 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.830619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-podinfo\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.931975 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.931881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-data\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.931975 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.931923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-certs\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.931975 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.931947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nll\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-kube-api-access-98nll\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.931982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-socket\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-podinfo\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-token\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1085211a-36ba-4063-a3a3-51db5f20c16b-istiod-ca-cert\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-credential-socket\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-envoy\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932368    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-certs\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932457    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-socket\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932978 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-credential-socket\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932978 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-data\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.932978 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.932932    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1085211a-36ba-4063-a3a3-51db5f20c16b-istiod-ca-cert\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.934429 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.934408    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-envoy\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.934662 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.934644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-podinfo\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.939891 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.939869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-token\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.940187 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.940168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nll\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-kube-api-access-98nll\") pod \"e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") " pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:41.980022 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:41.979994    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:42.261607 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:42.261582    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"]
Sep 11 01:13:42.263275 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:13:42.263246    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1085211a_36ba_4063_a3a3_51db5f20c16b.slice/crio-072749b6fa64141945a8addf75ed8f0b02febddef947c715054b7a828f475489 WatchSource:0}: Error finding container 072749b6fa64141945a8addf75ed8f0b02febddef947c715054b7a828f475489: Status 404 returned error can't find the container with id 072749b6fa64141945a8addf75ed8f0b02febddef947c715054b7a828f475489
Sep 11 01:13:42.265790 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:42.265750    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:42.265879 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:42.265837    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:42.265917 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:42.265884    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:13:42.587418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:42.587382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" event={"ID":"1085211a-36ba-4063-a3a3-51db5f20c16b","Type":"ContainerStarted","Data":"fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa"}
Sep 11 01:13:42.587562 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:42.587447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" event={"ID":"1085211a-36ba-4063-a3a3-51db5f20c16b","Type":"ContainerStarted","Data":"072749b6fa64141945a8addf75ed8f0b02febddef947c715054b7a828f475489"}
Sep 11 01:13:42.606628 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:42.606583    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" podStartSLOduration=1.606567798 podStartE2EDuration="1.606567798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:13:42.605381922 +0000 UTC m=+952.142033118" watchObservedRunningTime="2026-09-11 01:13:42.606567798 +0000 UTC m=+952.143218993"
Sep 11 01:13:42.980960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:42.980881    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:43.986739 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:43.986708    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:44.597953 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:44.597919    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:44.598844 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:44.598821    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:13:45.896913 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:45.896882    2583 patch_prober.go:28] interesting pod/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 11 01:13:45.897268 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:45.896960    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 11 01:13:46.089857 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.089829    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:46.272634 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272544    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvtjw\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-kube-api-access-wvtjw\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.272634 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272602    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-socket\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.272634 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272631    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-credential-socket\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.272904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272646    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-token\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.272904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272713    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-certs\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.272904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65588002-1822-4ca3-b8cd-001ee6ff2a72-istiod-ca-cert\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.272904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-envoy\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.272904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272875    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-podinfo\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.273144 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272937    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-data\") pod \"65588002-1822-4ca3-b8cd-001ee6ff2a72\" (UID: \"65588002-1822-4ca3-b8cd-001ee6ff2a72\") "
Sep 11 01:13:46.273144 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.272954    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:46.273144 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.273096    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:46.273297 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.273174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:46.273297 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.273235    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-data" (OuterVolumeSpecName: "istio-data") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:46.273433 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.273346    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.273433 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.273367    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.273433 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.273365    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65588002-1822-4ca3-b8cd-001ee6ff2a72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:13:46.273433 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.273381    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.273433 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.273397    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.274932 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.274901    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-kube-api-access-wvtjw" (OuterVolumeSpecName: "kube-api-access-wvtjw") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "kube-api-access-wvtjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:13:46.275098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.275080    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:13:46.275346 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.275081    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-token" (OuterVolumeSpecName: "istio-token") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:13:46.275346 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.275185    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65588002-1822-4ca3-b8cd-001ee6ff2a72" (UID: "65588002-1822-4ca3-b8cd-001ee6ff2a72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:13:46.373870 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.373842    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65588002-1822-4ca3-b8cd-001ee6ff2a72-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.373870 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.373868    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.374049 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.373879    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.374049 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.373888    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvtjw\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-kube-api-access-wvtjw\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.374049 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.373896    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65588002-1822-4ca3-b8cd-001ee6ff2a72-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:13:46.609474 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.609443    2583 generic.go:356] "Generic (PLEG): container finished" podID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerID="e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e" exitCode=0
Sep 11 01:13:46.609628 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.609490    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"
Sep 11 01:13:46.609628 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.609505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" event={"ID":"65588002-1822-4ca3-b8cd-001ee6ff2a72","Type":"ContainerDied","Data":"e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e"}
Sep 11 01:13:46.609628 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.609561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv" event={"ID":"65588002-1822-4ca3-b8cd-001ee6ff2a72","Type":"ContainerDied","Data":"8349f94e810e1aa4ca3b7623369e60d6dbf5d9568bc265519bc69178e36aefed"}
Sep 11 01:13:46.609628 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.609585    2583 scope.go:117] "RemoveContainer" containerID="e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e"
Sep 11 01:13:46.618858 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.618843    2583 scope.go:117] "RemoveContainer" containerID="99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50"
Sep 11 01:13:46.627237 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.627216    2583 scope.go:117] "RemoveContainer" containerID="e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e"
Sep 11 01:13:46.627630 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:13:46.627611    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e\": container with ID starting with e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e not found: ID does not exist" containerID="e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e"
Sep 11 01:13:46.627688 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.627640    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e"} err="failed to get container status \"e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e\": rpc error: code = NotFound desc = could not find container \"e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e\": container with ID starting with e74fb7fe56796d0b5a9fdac70992fc46f070e42ca7d9a1a7b24b8c3d244bd73e not found: ID does not exist"
Sep 11 01:13:46.627688 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.627656    2583 scope.go:117] "RemoveContainer" containerID="99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50"
Sep 11 01:13:46.627914 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:13:46.627899    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50\": container with ID starting with 99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50 not found: ID does not exist" containerID="99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50"
Sep 11 01:13:46.627954 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.627917    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50"} err="failed to get container status \"99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50\": rpc error: code = NotFound desc = could not find container \"99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50\": container with ID starting with 99660ab4c344b4ad83390a9291de40181824710ae9fdc611bd9a9314932f0c50 not found: ID does not exist"
Sep 11 01:13:46.633406 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.633387    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"]
Sep 11 01:13:46.637180 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:46.637160    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0b6ae428-gw-openshift-default-66845cc4d-h4kjv"]
Sep 11 01:13:47.037063 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:47.036988    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" path="/var/lib/kubelet/pods/65588002-1822-4ca3-b8cd-001ee6ff2a72/volumes"
Sep 11 01:13:47.478397 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:13:47.478365    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:14:04.778892 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:04.778808    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:14:07.717512 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:07.717478    2583 generic.go:356] "Generic (PLEG): container finished" podID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerID="fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa" exitCode=0
Sep 11 01:14:07.717928 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:07.717528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" event={"ID":"1085211a-36ba-4063-a3a3-51db5f20c16b","Type":"ContainerDied","Data":"fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa"}
Sep 11 01:14:07.717928 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:07.717907    2583 scope.go:117] "RemoveContainer" containerID="fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa"
Sep 11 01:14:07.719144 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:07.719114    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:14:07.719248 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:07.719198    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:14:07.719332 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:07.719250    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:14:08.726603 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.726558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" event={"ID":"1085211a-36ba-4063-a3a3-51db5f20c16b","Type":"ContainerStarted","Data":"ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168"}
Sep 11 01:14:08.981062 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.980984    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:14:08.981207 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.981131    2583 patch_prober.go:28] interesting pod/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 01:14:08.981207 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.981176    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 01:14:08.988068 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.988041    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"]
Sep 11 01:14:08.988435 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.988422    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy"
Sep 11 01:14:08.988481 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.988437    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy"
Sep 11 01:14:08.988481 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.988451    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy"
Sep 11 01:14:08.988481 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.988457    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy"
Sep 11 01:14:08.988573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.988517    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy"
Sep 11 01:14:08.988573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.988526    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65588002-1822-4ca3-b8cd-001ee6ff2a72" containerName="istio-proxy"
Sep 11 01:14:08.990668 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.990655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:08.992826 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.992803    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2c1e69\"/\"kube-root-ca.crt\""
Sep 11 01:14:08.993611 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.993589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2c1e69\"/\"default-dockercfg-g4bgz\""
Sep 11 01:14:08.993739 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.993625    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2c1e69\"/\"autores-model-2c1e69-kserve-self-signed-certs\""
Sep 11 01:14:08.993739 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.993628    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2c1e69\"/\"openshift-service-ca.crt\""
Sep 11 01:14:08.999825 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:08.999799    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"]
Sep 11 01:14:09.103817 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.103785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-model-cache\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.103817 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.103819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrd5\" (UniqueName: \"kubernetes.io/projected/8252728e-bba1-4f2e-a930-9d45525110b6-kube-api-access-7nrd5\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.104026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.103845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252728e-bba1-4f2e-a930-9d45525110b6-tls-certs\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.104026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.103923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-home\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.104026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.103977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-tmp-dir\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.104026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.104009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-dshm\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.205527 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.205490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-tmp-dir\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.205719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.205536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-dshm\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.205719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.205596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-model-cache\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.205719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.205614    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrd5\" (UniqueName: \"kubernetes.io/projected/8252728e-bba1-4f2e-a930-9d45525110b6-kube-api-access-7nrd5\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.205901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.205728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252728e-bba1-4f2e-a930-9d45525110b6-tls-certs\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.205901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.205793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-home\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.206004 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.205943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-tmp-dir\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.206004 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.205996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-model-cache\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.206131 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.206115    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-home\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.208022 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.207999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-dshm\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.208181 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.208166    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252728e-bba1-4f2e-a930-9d45525110b6-tls-certs\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.213317 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.213276    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrd5\" (UniqueName: \"kubernetes.io/projected/8252728e-bba1-4f2e-a930-9d45525110b6-kube-api-access-7nrd5\") pod \"autores-model-2c1e69-kserve-5478798587-gqch5\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") " pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.300905 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.300817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:09.601196 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.601171    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"]
Sep 11 01:14:09.603137 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:14:09.603116    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8252728e_bba1_4f2e_a930_9d45525110b6.slice/crio-b7f187d404f805b9443c3abd1013415ff68fdb7e983b679a6271055f6df47dc8 WatchSource:0}: Error finding container b7f187d404f805b9443c3abd1013415ff68fdb7e983b679a6271055f6df47dc8: Status 404 returned error can't find the container with id b7f187d404f805b9443c3abd1013415ff68fdb7e983b679a6271055f6df47dc8
Sep 11 01:14:09.738734 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.738557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5" event={"ID":"8252728e-bba1-4f2e-a930-9d45525110b6","Type":"ContainerStarted","Data":"d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10"}
Sep 11 01:14:09.738734 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.738602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5" event={"ID":"8252728e-bba1-4f2e-a930-9d45525110b6","Type":"ContainerStarted","Data":"b7f187d404f805b9443c3abd1013415ff68fdb7e983b679a6271055f6df47dc8"}
Sep 11 01:14:09.759928 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.759884    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5" podStartSLOduration=1.75986877 podStartE2EDuration="1.75986877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:14:09.758017 +0000 UTC m=+979.294668201" watchObservedRunningTime="2026-09-11 01:14:09.75986877 +0000 UTC m=+979.296519971"
Sep 11 01:14:09.981002 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.980916    2583 patch_prober.go:28] interesting pod/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 01:14:09.981002 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:09.980969    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 01:14:10.980673 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:10.980639    2583 patch_prober.go:28] interesting pod/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 01:14:10.981051 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:10.980689    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 01:14:11.980783 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:11.980747    2583 patch_prober.go:28] interesting pod/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 01:14:11.981154 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:11.980804    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 01:14:12.981201 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:12.981162    2583 patch_prober.go:28] interesting pod/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 11 01:14:12.981694 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:12.981225    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 11 01:14:13.984718 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:13.984687    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:14:13.985108 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:13.984972    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:14:13.985706 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:13.985686    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:14:17.185808 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:17.185767    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"]
Sep 11 01:14:17.186256 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:17.185991    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy" containerID="cri-o://ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168" gracePeriod=30
Sep 11 01:14:19.301619 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:19.301587    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:19.302005 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:19.301666    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:19.314463 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:19.314439    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:19.816823 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:19.816792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:14:21.293234 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.293193    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"]
Sep 11 01:14:21.296376 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.296355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.298647 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.298627    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6573d049-serving-cert\""
Sep 11 01:14:21.308419 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.308396    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"]
Sep 11 01:14:21.428752 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.428718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c015e2-bada-4e18-aff5-4f9c3701e609-openshift-service-ca\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.428913 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.428759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjzn\" (UniqueName: \"kubernetes.io/projected/d9c015e2-bada-4e18-aff5-4f9c3701e609-kube-api-access-xmjzn\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.428913 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.428876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-metrics-certs\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.428988 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.428932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.529666 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.529631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-metrics-certs\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.529826 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.529679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.529826 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.529719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c015e2-bada-4e18-aff5-4f9c3701e609-openshift-service-ca\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.529826 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.529743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjzn\" (UniqueName: \"kubernetes.io/projected/d9c015e2-bada-4e18-aff5-4f9c3701e609-kube-api-access-xmjzn\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.529930 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:14:21.529857    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-serving-cert: secret "maas-api-e2e-derive-6573d049-serving-cert" not found
Sep 11 01:14:21.529930 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:14:21.529923    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls podName:d9c015e2-bada-4e18-aff5-4f9c3701e609 nodeName:}" failed. No retries permitted until 2026-09-11 01:14:22.029902809 +0000 UTC m=+991.566554001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls") pod "maas-api-e2e-derive-6573d049-659d6788bf-8tvfz" (UID: "d9c015e2-bada-4e18-aff5-4f9c3701e609") : secret "maas-api-e2e-derive-6573d049-serving-cert" not found
Sep 11 01:14:21.530422 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.530398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c015e2-bada-4e18-aff5-4f9c3701e609-openshift-service-ca\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.532033 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.532007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-metrics-certs\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:21.538209 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:21.538186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjzn\" (UniqueName: \"kubernetes.io/projected/d9c015e2-bada-4e18-aff5-4f9c3701e609-kube-api-access-xmjzn\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:22.035046 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.035012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:22.035214 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:14:22.035125    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-serving-cert: secret "maas-api-e2e-derive-6573d049-serving-cert" not found
Sep 11 01:14:22.035214 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:14:22.035180    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls podName:d9c015e2-bada-4e18-aff5-4f9c3701e609 nodeName:}" failed. No retries permitted until 2026-09-11 01:14:23.035167702 +0000 UTC m=+992.571818893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls") pod "maas-api-e2e-derive-6573d049-659d6788bf-8tvfz" (UID: "d9c015e2-bada-4e18-aff5-4f9c3701e609") : secret "maas-api-e2e-derive-6573d049-serving-cert" not found
Sep 11 01:14:22.447077 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.447052    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:14:22.641128 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641098    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-token\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641128 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-credential-socket\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641159    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-certs\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641179    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-podinfo\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641200    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-data\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641269    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1085211a-36ba-4063-a3a3-51db5f20c16b-istiod-ca-cert\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98nll\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-kube-api-access-98nll\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641396    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-envoy\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641743 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641424    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-socket\") pod \"1085211a-36ba-4063-a3a3-51db5f20c16b\" (UID: \"1085211a-36ba-4063-a3a3-51db5f20c16b\") "
Sep 11 01:14:22.641743 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641463    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:14:22.641743 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641489    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:14:22.641743 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641563    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-data" (OuterVolumeSpecName: "istio-data") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:14:22.641961 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641756    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.641961 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641776    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.641961 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641781    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:14:22.641961 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641790    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.641961 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.641815    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1085211a-36ba-4063-a3a3-51db5f20c16b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:14:22.643452 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.643426    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-token" (OuterVolumeSpecName: "istio-token") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:14:22.643555 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.643475    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:14:22.643555 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.643502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:14:22.643555 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.643518    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-kube-api-access-98nll" (OuterVolumeSpecName: "kube-api-access-98nll") pod "1085211a-36ba-4063-a3a3-51db5f20c16b" (UID: "1085211a-36ba-4063-a3a3-51db5f20c16b"). InnerVolumeSpecName "kube-api-access-98nll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:14:22.742465 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.742438    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1085211a-36ba-4063-a3a3-51db5f20c16b-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.742465 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.742464    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98nll\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-kube-api-access-98nll\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.742465 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.742475    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.742728 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.742485    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1085211a-36ba-4063-a3a3-51db5f20c16b-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.742728 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.742497    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.742728 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.742510    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1085211a-36ba-4063-a3a3-51db5f20c16b-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:22.820187 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.820156    2583 generic.go:356] "Generic (PLEG): container finished" podID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerID="ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168" exitCode=0
Sep 11 01:14:22.820385 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.820197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" event={"ID":"1085211a-36ba-4063-a3a3-51db5f20c16b","Type":"ContainerDied","Data":"ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168"}
Sep 11 01:14:22.820385 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.820217    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"
Sep 11 01:14:22.820385 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.820248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b" event={"ID":"1085211a-36ba-4063-a3a3-51db5f20c16b","Type":"ContainerDied","Data":"072749b6fa64141945a8addf75ed8f0b02febddef947c715054b7a828f475489"}
Sep 11 01:14:22.820385 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.820279    2583 scope.go:117] "RemoveContainer" containerID="ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168"
Sep 11 01:14:22.829828 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.829735    2583 scope.go:117] "RemoveContainer" containerID="fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa"
Sep 11 01:14:22.837759 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.837741    2583 scope.go:117] "RemoveContainer" containerID="ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168"
Sep 11 01:14:22.838020 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:14:22.838001    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168\": container with ID starting with ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168 not found: ID does not exist" containerID="ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168"
Sep 11 01:14:22.838058 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.838029    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168"} err="failed to get container status \"ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168\": rpc error: code = NotFound desc = could not find container \"ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168\": container with ID starting with ab469f3cc565b719dc4a1ea901ce3997f3297a2863d1c19f03fc18c2a9a02168 not found: ID does not exist"
Sep 11 01:14:22.838058 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.838049    2583 scope.go:117] "RemoveContainer" containerID="fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa"
Sep 11 01:14:22.838357 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:14:22.838331    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa\": container with ID starting with fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa not found: ID does not exist" containerID="fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa"
Sep 11 01:14:22.838418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.838367    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa"} err="failed to get container status \"fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa\": rpc error: code = NotFound desc = could not find container \"fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa\": container with ID starting with fd397affc8523740a706beeee1d3ef5aa4ebcbfc644398fbd0baab5be69419fa not found: ID does not exist"
Sep 11 01:14:22.843025 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.843004    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"]
Sep 11 01:14:22.849893 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:22.849873    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9516b91d-openshift-default-76cb6c78d6-q9c5b"]
Sep 11 01:14:23.038010 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:23.037919    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" path="/var/lib/kubelet/pods/1085211a-36ba-4063-a3a3-51db5f20c16b/volumes"
Sep 11 01:14:23.044795 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:23.044766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:23.047192 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:23.047169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls\") pod \"maas-api-e2e-derive-6573d049-659d6788bf-8tvfz\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:23.109773 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:23.109746    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:23.380901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:23.380875    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"]
Sep 11 01:14:23.382626 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:14:23.382605    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c015e2_bada_4e18_aff5_4f9c3701e609.slice/crio-b63120fc800700a2fc1152fba3b9b65b458ba5e8aadbbafb71f2033088b18e7f WatchSource:0}: Error finding container b63120fc800700a2fc1152fba3b9b65b458ba5e8aadbbafb71f2033088b18e7f: Status 404 returned error can't find the container with id b63120fc800700a2fc1152fba3b9b65b458ba5e8aadbbafb71f2033088b18e7f
Sep 11 01:14:23.830428 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:23.830397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz" event={"ID":"d9c015e2-bada-4e18-aff5-4f9c3701e609","Type":"ContainerStarted","Data":"b63120fc800700a2fc1152fba3b9b65b458ba5e8aadbbafb71f2033088b18e7f"}
Sep 11 01:14:24.837828 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:24.837788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz" event={"ID":"d9c015e2-bada-4e18-aff5-4f9c3701e609","Type":"ContainerStarted","Data":"93f5e65b143ded3c4213805523310d383c5da07632f2226771bb3c7734d51b37"}
Sep 11 01:14:24.838234 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:24.838013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:24.856126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:24.856084    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz" podStartSLOduration=3.473164626 podStartE2EDuration="3.856068666s" totalImagesPullingTime="382.90404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:14:24.853791971 +0000 UTC m=+994.390443172" watchObservedRunningTime="2026-09-11 01:14:24.856068666 +0000 UTC m=+994.392719911"
Sep 11 01:14:27.682565 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:27.682525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:14:30.847888 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:30.847859    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:46.656021 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:46.655988    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"]
Sep 11 01:14:48.918523 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:48.918489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:14:51.817760 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:51.817727    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"]
Sep 11 01:14:51.818575 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:51.818537    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz" podUID="d9c015e2-bada-4e18-aff5-4f9c3701e609" containerName="maas-api" containerID="cri-o://93f5e65b143ded3c4213805523310d383c5da07632f2226771bb3c7734d51b37" gracePeriod=30
Sep 11 01:14:51.973259 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:51.973228    2583 generic.go:356] "Generic (PLEG): container finished" podID="d9c015e2-bada-4e18-aff5-4f9c3701e609" containerID="93f5e65b143ded3c4213805523310d383c5da07632f2226771bb3c7734d51b37" exitCode=0
Sep 11 01:14:51.973442 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:51.973298    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz" event={"ID":"d9c015e2-bada-4e18-aff5-4f9c3701e609","Type":"ContainerDied","Data":"93f5e65b143ded3c4213805523310d383c5da07632f2226771bb3c7734d51b37"}
Sep 11 01:14:52.072268 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.072213    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:52.201287 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.201250    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls\") pod \"d9c015e2-bada-4e18-aff5-4f9c3701e609\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") "
Sep 11 01:14:52.201479 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.201297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c015e2-bada-4e18-aff5-4f9c3701e609-openshift-service-ca\") pod \"d9c015e2-bada-4e18-aff5-4f9c3701e609\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") "
Sep 11 01:14:52.201479 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.201334    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-metrics-certs\") pod \"d9c015e2-bada-4e18-aff5-4f9c3701e609\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") "
Sep 11 01:14:52.201479 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.201443    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmjzn\" (UniqueName: \"kubernetes.io/projected/d9c015e2-bada-4e18-aff5-4f9c3701e609-kube-api-access-xmjzn\") pod \"d9c015e2-bada-4e18-aff5-4f9c3701e609\" (UID: \"d9c015e2-bada-4e18-aff5-4f9c3701e609\") "
Sep 11 01:14:52.201712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.201687    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c015e2-bada-4e18-aff5-4f9c3701e609-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9c015e2-bada-4e18-aff5-4f9c3701e609" (UID: "d9c015e2-bada-4e18-aff5-4f9c3701e609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:14:52.201865 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.201845    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c015e2-bada-4e18-aff5-4f9c3701e609-openshift-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:52.203653 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.203621    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d9c015e2-bada-4e18-aff5-4f9c3701e609" (UID: "d9c015e2-bada-4e18-aff5-4f9c3701e609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:14:52.203757 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.203678    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c015e2-bada-4e18-aff5-4f9c3701e609-kube-api-access-xmjzn" (OuterVolumeSpecName: "kube-api-access-xmjzn") pod "d9c015e2-bada-4e18-aff5-4f9c3701e609" (UID: "d9c015e2-bada-4e18-aff5-4f9c3701e609"). InnerVolumeSpecName "kube-api-access-xmjzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:14:52.203909 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.203885    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9c015e2-bada-4e18-aff5-4f9c3701e609" (UID: "d9c015e2-bada-4e18-aff5-4f9c3701e609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:14:52.302470 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.302442    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-maas-api-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:52.302470 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.302467    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c015e2-bada-4e18-aff5-4f9c3701e609-metrics-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:52.302470 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.302476    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjzn\" (UniqueName: \"kubernetes.io/projected/d9c015e2-bada-4e18-aff5-4f9c3701e609-kube-api-access-xmjzn\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:52.980024 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.979996    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"
Sep 11 01:14:52.980571 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.980033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz" event={"ID":"d9c015e2-bada-4e18-aff5-4f9c3701e609","Type":"ContainerDied","Data":"b63120fc800700a2fc1152fba3b9b65b458ba5e8aadbbafb71f2033088b18e7f"}
Sep 11 01:14:52.980571 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:52.980096    2583 scope.go:117] "RemoveContainer" containerID="93f5e65b143ded3c4213805523310d383c5da07632f2226771bb3c7734d51b37"
Sep 11 01:14:53.017391 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.017369    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"]
Sep 11 01:14:53.030285 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.030262    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6573d049-659d6788bf-8tvfz"]
Sep 11 01:14:53.036747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.036726    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c015e2-bada-4e18-aff5-4f9c3701e609" path="/var/lib/kubelet/pods/d9c015e2-bada-4e18-aff5-4f9c3701e609/volumes"
Sep 11 01:14:53.144990 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.144954    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"]
Sep 11 01:14:53.145228 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.145201    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb" podUID="e8347d4c-111e-48d5-aa31-11bc0f188ff9" containerName="maas-api" containerID="cri-o://6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c" gracePeriod=30
Sep 11 01:14:53.192086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.192052    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb" podUID="e8347d4c-111e-48d5-aa31-11bc0f188ff9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.44:8443/health\": dial tcp 10.132.0.44:8443: connect: connection refused"
Sep 11 01:14:53.386096 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.386074    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:14:53.511754 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.511686    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2wfx\" (UniqueName: \"kubernetes.io/projected/e8347d4c-111e-48d5-aa31-11bc0f188ff9-kube-api-access-h2wfx\") pod \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") "
Sep 11 01:14:53.511754 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.511726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-metrics-certs\") pod \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") "
Sep 11 01:14:53.511953 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.511788    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8347d4c-111e-48d5-aa31-11bc0f188ff9-openshift-service-ca\") pod \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") "
Sep 11 01:14:53.511953 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.511900    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls\") pod \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\" (UID: \"e8347d4c-111e-48d5-aa31-11bc0f188ff9\") "
Sep 11 01:14:53.512228 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.512199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8347d4c-111e-48d5-aa31-11bc0f188ff9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8347d4c-111e-48d5-aa31-11bc0f188ff9" (UID: "e8347d4c-111e-48d5-aa31-11bc0f188ff9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:14:53.513950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.513922    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8347d4c-111e-48d5-aa31-11bc0f188ff9" (UID: "e8347d4c-111e-48d5-aa31-11bc0f188ff9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:14:53.513950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.513929    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8347d4c-111e-48d5-aa31-11bc0f188ff9-kube-api-access-h2wfx" (OuterVolumeSpecName: "kube-api-access-h2wfx") pod "e8347d4c-111e-48d5-aa31-11bc0f188ff9" (UID: "e8347d4c-111e-48d5-aa31-11bc0f188ff9"). InnerVolumeSpecName "kube-api-access-h2wfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:14:53.514075 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.513962    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8347d4c-111e-48d5-aa31-11bc0f188ff9" (UID: "e8347d4c-111e-48d5-aa31-11bc0f188ff9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:14:53.613098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.613052    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-metrics-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:53.613098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.613090    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8347d4c-111e-48d5-aa31-11bc0f188ff9-openshift-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:53.613098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.613103    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8347d4c-111e-48d5-aa31-11bc0f188ff9-maas-api-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:53.613391 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.613118    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2wfx\" (UniqueName: \"kubernetes.io/projected/e8347d4c-111e-48d5-aa31-11bc0f188ff9-kube-api-access-h2wfx\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:14:53.986209 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.986173    2583 generic.go:356] "Generic (PLEG): container finished" podID="e8347d4c-111e-48d5-aa31-11bc0f188ff9" containerID="6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c" exitCode=0
Sep 11 01:14:53.986707 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.986226    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"
Sep 11 01:14:53.986707 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.986254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb" event={"ID":"e8347d4c-111e-48d5-aa31-11bc0f188ff9","Type":"ContainerDied","Data":"6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c"}
Sep 11 01:14:53.986707 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.986327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb" event={"ID":"e8347d4c-111e-48d5-aa31-11bc0f188ff9","Type":"ContainerDied","Data":"c35f2c609dff522417775a3f3b9060e66d1f92f9e5f59f89951abf5d1e107a5c"}
Sep 11 01:14:53.986707 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.986361    2583 scope.go:117] "RemoveContainer" containerID="6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c"
Sep 11 01:14:53.995586 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.995570    2583 scope.go:117] "RemoveContainer" containerID="6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c"
Sep 11 01:14:53.995852 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:14:53.995834    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c\": container with ID starting with 6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c not found: ID does not exist" containerID="6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c"
Sep 11 01:14:53.995906 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:53.995862    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c"} err="failed to get container status \"6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c\": rpc error: code = NotFound desc = could not find container \"6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c\": container with ID starting with 6665801e1feed86cae64091435ad0c754ff4d0f52eed798f0388aceca2e0217c not found: ID does not exist"
Sep 11 01:14:54.008709 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:54.008690    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"]
Sep 11 01:14:54.011573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:54.011554    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2c1e69-6f7d6586c5-k86lb"]
Sep 11 01:14:55.036781 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:14:55.036752    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8347d4c-111e-48d5-aa31-11bc0f188ff9" path="/var/lib/kubelet/pods/e8347d4c-111e-48d5-aa31-11bc0f188ff9/volumes"
Sep 11 01:15:00.217654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.217615    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"]
Sep 11 01:15:00.218057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.217993    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8347d4c-111e-48d5-aa31-11bc0f188ff9" containerName="maas-api"
Sep 11 01:15:00.218057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218005    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8347d4c-111e-48d5-aa31-11bc0f188ff9" containerName="maas-api"
Sep 11 01:15:00.218057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218013    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy"
Sep 11 01:15:00.218057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218018    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy"
Sep 11 01:15:00.218057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218023    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c015e2-bada-4e18-aff5-4f9c3701e609" containerName="maas-api"
Sep 11 01:15:00.218057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218029    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c015e2-bada-4e18-aff5-4f9c3701e609" containerName="maas-api"
Sep 11 01:15:00.218057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218038    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy"
Sep 11 01:15:00.218057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218043    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy"
Sep 11 01:15:00.218349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218139    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c015e2-bada-4e18-aff5-4f9c3701e609" containerName="maas-api"
Sep 11 01:15:00.218349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218148    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy"
Sep 11 01:15:00.218349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.218154    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8347d4c-111e-48d5-aa31-11bc0f188ff9" containerName="maas-api"
Sep 11 01:15:00.220923 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.220900    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"
Sep 11 01:15:00.223231 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.223205    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8"]
Sep 11 01:15:00.223689 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.223669    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1085211a-36ba-4063-a3a3-51db5f20c16b" containerName="istio-proxy"
Sep 11 01:15:00.224153 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.224133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q6gbn\""
Sep 11 01:15:00.226054 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.226034    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8"
Sep 11 01:15:00.232444 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.232425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"
Sep 11 01:15:00.238886 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.238867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8"
Sep 11 01:15:00.243912 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.243882    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8"]
Sep 11 01:15:00.257544 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.255544    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"]
Sep 11 01:15:00.517659 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.517634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"]
Sep 11 01:15:00.520037 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:15:00.520009    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99348cc8_24e0_4db1_a390_d0d97c57e195.slice/crio-f5c7e7c9bb166db63d9f38fe1e4b1aabe2eaffb0684ddadcc8ad816914e871dc WatchSource:0}: Error finding container f5c7e7c9bb166db63d9f38fe1e4b1aabe2eaffb0684ddadcc8ad816914e871dc: Status 404 returned error can't find the container with id f5c7e7c9bb166db63d9f38fe1e4b1aabe2eaffb0684ddadcc8ad816914e871dc
Sep 11 01:15:00.522378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.522354    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 01:15:00.549648 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:00.549625    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8"]
Sep 11 01:15:00.551202 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:15:00.551170    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b6112d_a4d8_4d40_93d3_b8bf0773cb4f.slice/crio-6a7c5985329fd4756a8fbd90342d6dbf3572e666bce38a38787640d0100bf418 WatchSource:0}: Error finding container 6a7c5985329fd4756a8fbd90342d6dbf3572e666bce38a38787640d0100bf418: Status 404 returned error can't find the container with id 6a7c5985329fd4756a8fbd90342d6dbf3572e666bce38a38787640d0100bf418
Sep 11 01:15:01.024773 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:01.024733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v" event={"ID":"99348cc8-24e0-4db1-a390-d0d97c57e195","Type":"ContainerStarted","Data":"f5c7e7c9bb166db63d9f38fe1e4b1aabe2eaffb0684ddadcc8ad816914e871dc"}
Sep 11 01:15:01.026245 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:01.026215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8" event={"ID":"77b6112d-a4d8-4d40-93d3-b8bf0773cb4f","Type":"ContainerStarted","Data":"6a7c5985329fd4756a8fbd90342d6dbf3572e666bce38a38787640d0100bf418"}
Sep 11 01:15:01.676589 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:01.676519    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5" podUID="8252728e-bba1-4f2e-a930-9d45525110b6" containerName="main" containerID="cri-o://d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10" gracePeriod=45
Sep 11 01:15:01.961890 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:01.961863    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:15:02.036530 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.036491    2583 generic.go:356] "Generic (PLEG): container finished" podID="8252728e-bba1-4f2e-a930-9d45525110b6" containerID="d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10" exitCode=0
Sep 11 01:15:02.036714 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.036658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5" event={"ID":"8252728e-bba1-4f2e-a930-9d45525110b6","Type":"ContainerDied","Data":"d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10"}
Sep 11 01:15:02.036714 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.036695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5" event={"ID":"8252728e-bba1-4f2e-a930-9d45525110b6","Type":"ContainerDied","Data":"b7f187d404f805b9443c3abd1013415ff68fdb7e983b679a6271055f6df47dc8"}
Sep 11 01:15:02.036838 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.036716    2583 scope.go:117] "RemoveContainer" containerID="d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10"
Sep 11 01:15:02.036891 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.036873    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"
Sep 11 01:15:02.048245 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.048220    2583 scope.go:117] "RemoveContainer" containerID="d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10"
Sep 11 01:15:02.048699 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:15:02.048624    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10\": container with ID starting with d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10 not found: ID does not exist" containerID="d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10"
Sep 11 01:15:02.048699 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.048669    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10"} err="failed to get container status \"d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10\": rpc error: code = NotFound desc = could not find container \"d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10\": container with ID starting with d3f22f020cfea43738e46f622a6625fc50c916ec3b61047bd97ecde5eeeb0a10 not found: ID does not exist"
Sep 11 01:15:02.091081 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.090961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-tmp-dir\") pod \"8252728e-bba1-4f2e-a930-9d45525110b6\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") "
Sep 11 01:15:02.091081 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252728e-bba1-4f2e-a930-9d45525110b6-tls-certs\") pod \"8252728e-bba1-4f2e-a930-9d45525110b6\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") "
Sep 11 01:15:02.091081 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nrd5\" (UniqueName: \"kubernetes.io/projected/8252728e-bba1-4f2e-a930-9d45525110b6-kube-api-access-7nrd5\") pod \"8252728e-bba1-4f2e-a930-9d45525110b6\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") "
Sep 11 01:15:02.091410 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091114    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-model-cache\") pod \"8252728e-bba1-4f2e-a930-9d45525110b6\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") "
Sep 11 01:15:02.091410 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-home\") pod \"8252728e-bba1-4f2e-a930-9d45525110b6\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") "
Sep 11 01:15:02.091410 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091221    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-dshm\") pod \"8252728e-bba1-4f2e-a930-9d45525110b6\" (UID: \"8252728e-bba1-4f2e-a930-9d45525110b6\") "
Sep 11 01:15:02.091410 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091266    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8252728e-bba1-4f2e-a930-9d45525110b6" (UID: "8252728e-bba1-4f2e-a930-9d45525110b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:02.091623 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091501    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "8252728e-bba1-4f2e-a930-9d45525110b6" (UID: "8252728e-bba1-4f2e-a930-9d45525110b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:02.091623 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091541    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-tmp-dir\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:15:02.091746 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.091660    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-home" (OuterVolumeSpecName: "home") pod "8252728e-bba1-4f2e-a930-9d45525110b6" (UID: "8252728e-bba1-4f2e-a930-9d45525110b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:02.094207 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.094093    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8252728e-bba1-4f2e-a930-9d45525110b6-kube-api-access-7nrd5" (OuterVolumeSpecName: "kube-api-access-7nrd5") pod "8252728e-bba1-4f2e-a930-9d45525110b6" (UID: "8252728e-bba1-4f2e-a930-9d45525110b6"). InnerVolumeSpecName "kube-api-access-7nrd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:15:02.094987 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.094332    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8252728e-bba1-4f2e-a930-9d45525110b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8252728e-bba1-4f2e-a930-9d45525110b6" (UID: "8252728e-bba1-4f2e-a930-9d45525110b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:15:02.094987 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.094857    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-dshm" (OuterVolumeSpecName: "dshm") pod "8252728e-bba1-4f2e-a930-9d45525110b6" (UID: "8252728e-bba1-4f2e-a930-9d45525110b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:15:02.192266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.192188    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252728e-bba1-4f2e-a930-9d45525110b6-tls-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:15:02.192266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.192219    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nrd5\" (UniqueName: \"kubernetes.io/projected/8252728e-bba1-4f2e-a930-9d45525110b6-kube-api-access-7nrd5\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:15:02.192266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.192231    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-model-cache\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:15:02.192266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.192243    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-home\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:15:02.192266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.192253    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8252728e-bba1-4f2e-a930-9d45525110b6-dshm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:15:02.359766 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.359736    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"]
Sep 11 01:15:02.364086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:02.364059    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-2c1e69/autores-model-2c1e69-kserve-5478798587-gqch5"]
Sep 11 01:15:03.037365 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:03.037333    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8252728e-bba1-4f2e-a930-9d45525110b6" path="/var/lib/kubelet/pods/8252728e-bba1-4f2e-a930-9d45525110b6/volumes"
Sep 11 01:15:04.050153 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:04.050110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v" event={"ID":"99348cc8-24e0-4db1-a390-d0d97c57e195","Type":"ContainerStarted","Data":"fc4af76ddb41a9c2c1162bc52f3965e89482fa59326cfd42148110e6393e1766"}
Sep 11 01:15:04.051630 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:04.051603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8" event={"ID":"77b6112d-a4d8-4d40-93d3-b8bf0773cb4f","Type":"ContainerStarted","Data":"497b3d61a0c2843fc79eba4edcc58f4e71108dfa52f2e5f1d9cb3820cadbc3ab"}
Sep 11 01:15:04.066416 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:04.066373    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v" podStartSLOduration=1.579056482 podStartE2EDuration="4.066360283s" totalImagesPullingTime="2.487303801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:04.064670353 +0000 UTC m=+1033.601321554" watchObservedRunningTime="2026-09-11 01:15:04.066360283 +0000 UTC m=+1033.603011484"
Sep 11 01:15:04.080902 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:04.080858    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8" podStartSLOduration=1.619398884 podStartE2EDuration="4.080832048s" totalImagesPullingTime="2.461433164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:04.079795185 +0000 UTC m=+1033.616446383" watchObservedRunningTime="2026-09-11 01:15:04.080832048 +0000 UTC m=+1033.617483249"
Sep 11 01:15:05.057433 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.057400    2583 generic.go:356] "Generic (PLEG): container finished" podID="99348cc8-24e0-4db1-a390-d0d97c57e195" containerID="fc4af76ddb41a9c2c1162bc52f3965e89482fa59326cfd42148110e6393e1766" exitCode=0
Sep 11 01:15:05.057883 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.057481    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v" event={"ID":"99348cc8-24e0-4db1-a390-d0d97c57e195","Type":"ContainerDied","Data":"fc4af76ddb41a9c2c1162bc52f3965e89482fa59326cfd42148110e6393e1766"}
Sep 11 01:15:05.059044 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.059010    2583 generic.go:356] "Generic (PLEG): container finished" podID="77b6112d-a4d8-4d40-93d3-b8bf0773cb4f" containerID="497b3d61a0c2843fc79eba4edcc58f4e71108dfa52f2e5f1d9cb3820cadbc3ab" exitCode=0
Sep 11 01:15:05.059164 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.059033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8" event={"ID":"77b6112d-a4d8-4d40-93d3-b8bf0773cb4f","Type":"ContainerDied","Data":"497b3d61a0c2843fc79eba4edcc58f4e71108dfa52f2e5f1d9cb3820cadbc3ab"}
Sep 11 01:15:05.770521 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.770300    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"]
Sep 11 01:15:05.771925 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.771117    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8252728e-bba1-4f2e-a930-9d45525110b6" containerName="main"
Sep 11 01:15:05.771925 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.771147    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8252728e-bba1-4f2e-a930-9d45525110b6" containerName="main"
Sep 11 01:15:05.771925 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.771378    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8252728e-bba1-4f2e-a930-9d45525110b6" containerName="main"
Sep 11 01:15:05.774458 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.774434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.776782 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.776758    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 01:15:05.776898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.776769    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-987d60-openshift-default-dockercfg-8tl5v\""
Sep 11 01:15:05.781665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.781645    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"]
Sep 11 01:15:05.924353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mrw\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-kube-api-access-x6mrw\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.924353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-podinfo\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.924573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-certs\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.924573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-token\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.924573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istiod-ca-cert\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.924573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-data\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.924740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-socket\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.924740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-credential-socket\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:05.924740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:05.924693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-envoy\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mrw\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-kube-api-access-x6mrw\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-podinfo\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-certs\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-token\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025563 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istiod-ca-cert\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025563 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-data\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025563 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-socket\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025563 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-credential-socket\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025563 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-envoy\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025880 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-certs\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025880 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-credential-socket\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.025880 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.025845    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-socket\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.026074 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.026053    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-data\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.026294 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.026277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istiod-ca-cert\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.027813 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.027789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-envoy\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.027941 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.027851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-podinfo\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.032377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.032356    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-token\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.032904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.032881    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mrw\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-kube-api-access-x6mrw\") pod \"e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") " pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.086778 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.086747    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:06.216197 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.216170    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8"
Sep 11 01:15:06.225442 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.225416    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"
Sep 11 01:15:06.369681 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.369646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"]
Sep 11 01:15:06.372437 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:15:06.372413    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a7c7ac_4e1c_440b_8864_d7a67160d8d9.slice/crio-c8922ded18840437c1828256420536abf061d26d256d64273dde9b2059f7e6bf WatchSource:0}: Error finding container c8922ded18840437c1828256420536abf061d26d256d64273dde9b2059f7e6bf: Status 404 returned error can't find the container with id c8922ded18840437c1828256420536abf061d26d256d64273dde9b2059f7e6bf
Sep 11 01:15:06.374775 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.374748    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:15:06.374837 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.374808    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:15:06.374837 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:06.374834    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:15:07.069576 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.069551    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"
Sep 11 01:15:07.069697 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.069554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v" event={"ID":"99348cc8-24e0-4db1-a390-d0d97c57e195","Type":"ContainerDied","Data":"f5c7e7c9bb166db63d9f38fe1e4b1aabe2eaffb0684ddadcc8ad816914e871dc"}
Sep 11 01:15:07.069758 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.069700    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5c7e7c9bb166db63d9f38fe1e4b1aabe2eaffb0684ddadcc8ad816914e871dc"
Sep 11 01:15:07.070921 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.070900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8" event={"ID":"77b6112d-a4d8-4d40-93d3-b8bf0773cb4f","Type":"ContainerDied","Data":"6a7c5985329fd4756a8fbd90342d6dbf3572e666bce38a38787640d0100bf418"}
Sep 11 01:15:07.070921 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.070911    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818155-6lft8"
Sep 11 01:15:07.071093 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.070938    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a7c5985329fd4756a8fbd90342d6dbf3572e666bce38a38787640d0100bf418"
Sep 11 01:15:07.072846 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.072825    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" event={"ID":"75a7c7ac-4e1c-440b-8864-d7a67160d8d9","Type":"ContainerStarted","Data":"7bd806b565b698df7c8ce35cf0f61d853381adc5a10a8363ce292d35ccc43beb"}
Sep 11 01:15:07.073012 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.072989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" event={"ID":"75a7c7ac-4e1c-440b-8864-d7a67160d8d9","Type":"ContainerStarted","Data":"c8922ded18840437c1828256420536abf061d26d256d64273dde9b2059f7e6bf"}
Sep 11 01:15:07.087703 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.087687    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:07.092704 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.092654    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podStartSLOduration=2.092641919 podStartE2EDuration="2.092641919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:07.090126578 +0000 UTC m=+1036.626777798" watchObservedRunningTime="2026-09-11 01:15:07.092641919 +0000 UTC m=+1036.629293116"
Sep 11 01:15:07.093958 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:07.093941    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:08.078739 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:08.078716    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:08.079759 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:08.079734    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:10.312499 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.312471    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"]
Sep 11 01:15:10.312904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.312890    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b6112d-a4d8-4d40-93d3-b8bf0773cb4f" containerName="cleanup"
Sep 11 01:15:10.312942 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.312906    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b6112d-a4d8-4d40-93d3-b8bf0773cb4f" containerName="cleanup"
Sep 11 01:15:10.312942 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.312925    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99348cc8-24e0-4db1-a390-d0d97c57e195" containerName="cleanup"
Sep 11 01:15:10.312942 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.312930    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="99348cc8-24e0-4db1-a390-d0d97c57e195" containerName="cleanup"
Sep 11 01:15:10.313048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.312987    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="99348cc8-24e0-4db1-a390-d0d97c57e195" containerName="cleanup"
Sep 11 01:15:10.313048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.313001    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b6112d-a4d8-4d40-93d3-b8bf0773cb4f" containerName="cleanup"
Sep 11 01:15:10.315545 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.315526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.317590 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.317572    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-987d60-serving-cert\""
Sep 11 01:15:10.327259 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.327239    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"]
Sep 11 01:15:10.467387 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.467350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-metrics-certs\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.467557 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.467486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-openshift-service-ca\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.467557 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.467536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-maas-api-tls\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.467665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.467563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkz2\" (UniqueName: \"kubernetes.io/projected/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-kube-api-access-qqkz2\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.568233 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.568151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-maas-api-tls\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.568233 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.568192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkz2\" (UniqueName: \"kubernetes.io/projected/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-kube-api-access-qqkz2\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.568233 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.568227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-metrics-certs\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.568468 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.568300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-openshift-service-ca\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.569005 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.568978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-openshift-service-ca\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.570579 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.570558    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-maas-api-tls\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.570749 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.570730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-metrics-certs\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.577323 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.577287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkz2\" (UniqueName: \"kubernetes.io/projected/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-kube-api-access-qqkz2\") pod \"maas-api-e2e-explicit-987d60-5cccb75994-xnd2k\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.631318 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.631285    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:10.931831 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:10.931781    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"]
Sep 11 01:15:10.933838 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:15:10.933810    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5624e6b3_30bf_4b3c_8f81_6b9eec75362b.slice/crio-14c2472933763b1dd22b67a30d00cb9b08a5e7ea580f0a65e6d0643ffe7a857d WatchSource:0}: Error finding container 14c2472933763b1dd22b67a30d00cb9b08a5e7ea580f0a65e6d0643ffe7a857d: Status 404 returned error can't find the container with id 14c2472933763b1dd22b67a30d00cb9b08a5e7ea580f0a65e6d0643ffe7a857d
Sep 11 01:15:11.094449 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.094409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k" event={"ID":"5624e6b3-30bf-4b3c-8f81-6b9eec75362b","Type":"ContainerStarted","Data":"14c2472933763b1dd22b67a30d00cb9b08a5e7ea580f0a65e6d0643ffe7a857d"}
Sep 11 01:15:11.221570 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.221485    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"]
Sep 11 01:15:11.224935 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.224914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:11.227254 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.227225    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-987d60-dockercfg-gchnz\""
Sep 11 01:15:11.227375 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.227257    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-987d60\""
Sep 11 01:15:11.235579 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.235555    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"]
Sep 11 01:15:11.374864 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.374840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcjg8\" (UniqueName: \"kubernetes.io/projected/1aee6c93-496b-4416-9b34-642d485819d4-kube-api-access-wcjg8\") pod \"payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z\" (UID: \"1aee6c93-496b-4416-9b34-642d485819d4\") " pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:11.375156 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.374907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee6c93-496b-4416-9b34-642d485819d4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z\" (UID: \"1aee6c93-496b-4416-9b34-642d485819d4\") " pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:11.476287 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.476203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcjg8\" (UniqueName: \"kubernetes.io/projected/1aee6c93-496b-4416-9b34-642d485819d4-kube-api-access-wcjg8\") pod \"payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z\" (UID: \"1aee6c93-496b-4416-9b34-642d485819d4\") " pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:11.476287 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.476273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee6c93-496b-4416-9b34-642d485819d4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z\" (UID: \"1aee6c93-496b-4416-9b34-642d485819d4\") " pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:11.476998 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.476978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee6c93-496b-4416-9b34-642d485819d4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z\" (UID: \"1aee6c93-496b-4416-9b34-642d485819d4\") " pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:11.483557 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.483531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcjg8\" (UniqueName: \"kubernetes.io/projected/1aee6c93-496b-4416-9b34-642d485819d4-kube-api-access-wcjg8\") pod \"payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z\" (UID: \"1aee6c93-496b-4416-9b34-642d485819d4\") " pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:11.537628 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.537598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:11.677424 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.677388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:15:11.812532 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:11.812506    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"]
Sep 11 01:15:11.813038 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:15:11.813019    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aee6c93_496b_4416_9b34_642d485819d4.slice/crio-2e840cad4f0865bce128194b8795c790f155add26da7dda16d9e6628eb30acff WatchSource:0}: Error finding container 2e840cad4f0865bce128194b8795c790f155add26da7dda16d9e6628eb30acff: Status 404 returned error can't find the container with id 2e840cad4f0865bce128194b8795c790f155add26da7dda16d9e6628eb30acff
Sep 11 01:15:12.101545 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:12.101441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z" event={"ID":"1aee6c93-496b-4416-9b34-642d485819d4","Type":"ContainerStarted","Data":"b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943"}
Sep 11 01:15:12.101545 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:12.101512    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:12.101545 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:12.101538    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z" event={"ID":"1aee6c93-496b-4416-9b34-642d485819d4","Type":"ContainerStarted","Data":"2e840cad4f0865bce128194b8795c790f155add26da7dda16d9e6628eb30acff"}
Sep 11 01:15:12.103041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:12.103019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k" event={"ID":"5624e6b3-30bf-4b3c-8f81-6b9eec75362b","Type":"ContainerStarted","Data":"2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76"}
Sep 11 01:15:12.103162 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:12.103147    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:12.118009 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:12.117966    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z" podStartSLOduration=1.117951806 podStartE2EDuration="1.117951806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:12.115507116 +0000 UTC m=+1041.652158316" watchObservedRunningTime="2026-09-11 01:15:12.117951806 +0000 UTC m=+1041.654603007"
Sep 11 01:15:12.132098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:12.132057    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k" podStartSLOduration=1.7406104980000001 podStartE2EDuration="2.132041885s" totalImagesPullingTime="391.431387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:12.130465755 +0000 UTC m=+1041.667116968" watchObservedRunningTime="2026-09-11 01:15:12.132041885 +0000 UTC m=+1041.668693119"
Sep 11 01:15:18.114014 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:18.113986    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:15:23.110507 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:23.110477    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:15:29.676534 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:29.676500    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:15:39.120502 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:39.120461    2583 patch_prober.go:28] interesting pod/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 01:15:39.121086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:39.120555    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 01:15:41.251968 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:41.251937    2583 generic.go:356] "Generic (PLEG): container finished" podID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerID="7bd806b565b698df7c8ce35cf0f61d853381adc5a10a8363ce292d35ccc43beb" exitCode=0
Sep 11 01:15:41.252431 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:41.252007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" event={"ID":"75a7c7ac-4e1c-440b-8864-d7a67160d8d9","Type":"ContainerDied","Data":"7bd806b565b698df7c8ce35cf0f61d853381adc5a10a8363ce292d35ccc43beb"}
Sep 11 01:15:41.252431 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:41.252382    2583 scope.go:117] "RemoveContainer" containerID="7bd806b565b698df7c8ce35cf0f61d853381adc5a10a8363ce292d35ccc43beb"
Sep 11 01:15:41.253558 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:41.253524    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:15:41.253671 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:41.253601    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:15:41.253671 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:41.253635    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:15:42.259268 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:42.259233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" event={"ID":"75a7c7ac-4e1c-440b-8864-d7a67160d8d9","Type":"ContainerStarted","Data":"ad59b38908005c1bf4e53786f036af4c391b1c1a56a9e4f3c119dd9435f01bdd"}
Sep 11 01:15:43.087347 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:43.087300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:43.087554 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:43.087485    2583 patch_prober.go:28] interesting pod/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:15:43.087632 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:43.087566    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:15:44.087332 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.087291    2583 patch_prober.go:28] interesting pod/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:15:44.087715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.087354    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:15:44.098773 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.098746    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"]
Sep 11 01:15:44.101242 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.101227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.106272 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.106248    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cd176e\"/\"openshift-service-ca.crt\""
Sep 11 01:15:44.106511 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.106495    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cd176e\"/\"default-dockercfg-kwbgx\""
Sep 11 01:15:44.106569 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.106501    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cd176e\"/\"premium-w4-cd176e-kserve-self-signed-certs\""
Sep 11 01:15:44.106783 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.106770    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cd176e\"/\"kube-root-ca.crt\""
Sep 11 01:15:44.136735 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.134880    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"]
Sep 11 01:15:44.180862 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.180835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-home\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.181035 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.180890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-dshm\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.181035 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.180912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5088ba-35be-425c-917f-4a3eb39094f9-tls-certs\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.181035 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.180932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-model-cache\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.181201 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.181122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-tmp-dir\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.181259 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.181231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pjj\" (UniqueName: \"kubernetes.io/projected/8e5088ba-35be-425c-917f-4a3eb39094f9-kube-api-access-r5pjj\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282193 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-tmp-dir\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282395 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pjj\" (UniqueName: \"kubernetes.io/projected/8e5088ba-35be-425c-917f-4a3eb39094f9-kube-api-access-r5pjj\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282395 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-home\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282395 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-dshm\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282395 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5088ba-35be-425c-917f-4a3eb39094f9-tls-certs\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282395 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-model-cache\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282720 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-model-cache\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-home\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.282853 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.282808    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-tmp-dir\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.284675 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.284656    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-dshm\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.285014 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.284999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5088ba-35be-425c-917f-4a3eb39094f9-tls-certs\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.290881 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.290858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pjj\" (UniqueName: \"kubernetes.io/projected/8e5088ba-35be-425c-917f-4a3eb39094f9-kube-api-access-r5pjj\") pod \"premium-w4-cd176e-kserve-c5b8ddb6-7d7t6\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") " pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.410930 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.410835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:44.710088 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:44.710055    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"]
Sep 11 01:15:44.711961 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:15:44.711928    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e5088ba_35be_425c_917f_4a3eb39094f9.slice/crio-af6292c82b78b21a4fee8c3f775e9f76c21973f53380d57f69833cf13ad0566a WatchSource:0}: Error finding container af6292c82b78b21a4fee8c3f775e9f76c21973f53380d57f69833cf13ad0566a: Status 404 returned error can't find the container with id af6292c82b78b21a4fee8c3f775e9f76c21973f53380d57f69833cf13ad0566a
Sep 11 01:15:45.088121 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:45.088086    2583 patch_prober.go:28] interesting pod/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:15:45.088515 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:45.088143    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:15:45.275517 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:45.275485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6" event={"ID":"8e5088ba-35be-425c-917f-4a3eb39094f9","Type":"ContainerStarted","Data":"640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682"}
Sep 11 01:15:45.275517 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:45.275524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6" event={"ID":"8e5088ba-35be-425c-917f-4a3eb39094f9","Type":"ContainerStarted","Data":"af6292c82b78b21a4fee8c3f775e9f76c21973f53380d57f69833cf13ad0566a"}
Sep 11 01:15:45.289909 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:45.289854    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6" podStartSLOduration=1.289835793 podStartE2EDuration="1.289835793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:45.289808728 +0000 UTC m=+1074.826459930" watchObservedRunningTime="2026-09-11 01:15:45.289835793 +0000 UTC m=+1074.826486995"
Sep 11 01:15:46.087727 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:46.087691    2583 patch_prober.go:28] interesting pod/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:15:46.087898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:46.087740    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:15:47.087808 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:47.087773    2583 patch_prober.go:28] interesting pod/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 01:15:47.088213 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:47.087830    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 01:15:48.090933 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:48.090903    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:48.091330 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:48.091179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:48.091933 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:48.091916    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:15:49.889835 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:49.889800    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:15:54.410950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:54.410913    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:54.411463 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:54.410967    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:54.425211 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:54.425180    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:15:55.347839 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:15:55.347810    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:16:21.593449 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:21.593409    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:16:36.642610 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.642579    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"]
Sep 11 01:16:36.645185 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.645169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:36.647481 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.647453    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ef3c90aa\""
Sep 11 01:16:36.647589 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.647487    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ef3c90aa-dockercfg-jdtrr\""
Sep 11 01:16:36.655496 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.655474    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"]
Sep 11 01:16:36.772545 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.772510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmgl\" (UniqueName: \"kubernetes.io/projected/1c99998c-f382-40c4-8dc3-617349afcea4-kube-api-access-htmgl\") pod \"payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b\" (UID: \"1c99998c-f382-40c4-8dc3-617349afcea4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:36.772711 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.772607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c99998c-f382-40c4-8dc3-617349afcea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b\" (UID: \"1c99998c-f382-40c4-8dc3-617349afcea4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:36.873672 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.873631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmgl\" (UniqueName: \"kubernetes.io/projected/1c99998c-f382-40c4-8dc3-617349afcea4-kube-api-access-htmgl\") pod \"payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b\" (UID: \"1c99998c-f382-40c4-8dc3-617349afcea4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:36.873869 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.873719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c99998c-f382-40c4-8dc3-617349afcea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b\" (UID: \"1c99998c-f382-40c4-8dc3-617349afcea4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:36.874441 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.874418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c99998c-f382-40c4-8dc3-617349afcea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b\" (UID: \"1c99998c-f382-40c4-8dc3-617349afcea4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:36.883214 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.883187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmgl\" (UniqueName: \"kubernetes.io/projected/1c99998c-f382-40c4-8dc3-617349afcea4-kube-api-access-htmgl\") pod \"payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b\" (UID: \"1c99998c-f382-40c4-8dc3-617349afcea4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:36.955442 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:36.955375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:37.229713 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:37.229682    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"]
Sep 11 01:16:37.231903 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:16:37.231871    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c99998c_f382_40c4_8dc3_617349afcea4.slice/crio-7ffa0b8927144d6ce45fab97331f42a33c57dc758b9348a91d2884cf5a29fca7 WatchSource:0}: Error finding container 7ffa0b8927144d6ce45fab97331f42a33c57dc758b9348a91d2884cf5a29fca7: Status 404 returned error can't find the container with id 7ffa0b8927144d6ce45fab97331f42a33c57dc758b9348a91d2884cf5a29fca7
Sep 11 01:16:37.555748 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:37.555715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b" event={"ID":"1c99998c-f382-40c4-8dc3-617349afcea4","Type":"ContainerStarted","Data":"245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0"}
Sep 11 01:16:37.555907 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:37.555767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b" event={"ID":"1c99998c-f382-40c4-8dc3-617349afcea4","Type":"ContainerStarted","Data":"7ffa0b8927144d6ce45fab97331f42a33c57dc758b9348a91d2884cf5a29fca7"}
Sep 11 01:16:37.555907 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:37.555807    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:37.574040 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:37.573992    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b" podStartSLOduration=1.573978691 podStartE2EDuration="1.573978691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:37.571464187 +0000 UTC m=+1127.108115390" watchObservedRunningTime="2026-09-11 01:16:37.573978691 +0000 UTC m=+1127.110629904"
Sep 11 01:16:48.563506 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:48.563471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:16:51.485911 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:16:51.485880    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:17:11.292719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:11.292684    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:17:17.663949 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:17.663869    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"]
Sep 11 01:17:17.664386 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:17.664138    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b" podUID="1c99998c-f382-40c4-8dc3-617349afcea4" containerName="payload-pre-processing" containerID="cri-o://245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0" gracePeriod=30
Sep 11 01:17:17.936694 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:17.936670    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:17:18.061071 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.061034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c99998c-f382-40c4-8dc3-617349afcea4-plugins-config-volume\") pod \"1c99998c-f382-40c4-8dc3-617349afcea4\" (UID: \"1c99998c-f382-40c4-8dc3-617349afcea4\") "
Sep 11 01:17:18.061258 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.061116    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htmgl\" (UniqueName: \"kubernetes.io/projected/1c99998c-f382-40c4-8dc3-617349afcea4-kube-api-access-htmgl\") pod \"1c99998c-f382-40c4-8dc3-617349afcea4\" (UID: \"1c99998c-f382-40c4-8dc3-617349afcea4\") "
Sep 11 01:17:18.061456 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.061432    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c99998c-f382-40c4-8dc3-617349afcea4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c99998c-f382-40c4-8dc3-617349afcea4" (UID: "1c99998c-f382-40c4-8dc3-617349afcea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:17:18.063129 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.063105    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c99998c-f382-40c4-8dc3-617349afcea4-kube-api-access-htmgl" (OuterVolumeSpecName: "kube-api-access-htmgl") pod "1c99998c-f382-40c4-8dc3-617349afcea4" (UID: "1c99998c-f382-40c4-8dc3-617349afcea4"). InnerVolumeSpecName "kube-api-access-htmgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:17:18.162778 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.162746    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c99998c-f382-40c4-8dc3-617349afcea4-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:17:18.162778 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.162773    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htmgl\" (UniqueName: \"kubernetes.io/projected/1c99998c-f382-40c4-8dc3-617349afcea4-kube-api-access-htmgl\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:17:18.764667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.764617    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c99998c-f382-40c4-8dc3-617349afcea4" containerID="245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0" exitCode=0
Sep 11 01:17:18.765090 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.764676    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"
Sep 11 01:17:18.765090 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.764698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b" event={"ID":"1c99998c-f382-40c4-8dc3-617349afcea4","Type":"ContainerDied","Data":"245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0"}
Sep 11 01:17:18.765090 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.764751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b" event={"ID":"1c99998c-f382-40c4-8dc3-617349afcea4","Type":"ContainerDied","Data":"7ffa0b8927144d6ce45fab97331f42a33c57dc758b9348a91d2884cf5a29fca7"}
Sep 11 01:17:18.765090 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.764793    2583 scope.go:117] "RemoveContainer" containerID="245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0"
Sep 11 01:17:18.774216 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.774202    2583 scope.go:117] "RemoveContainer" containerID="245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0"
Sep 11 01:17:18.774487 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:17:18.774470    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0\": container with ID starting with 245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0 not found: ID does not exist" containerID="245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0"
Sep 11 01:17:18.774534 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.774496    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0"} err="failed to get container status \"245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0\": rpc error: code = NotFound desc = could not find container \"245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0\": container with ID starting with 245737b08bfc1bca1c5b243b8f5b5981a8043de7db855766fc775b64c124a2a0 not found: ID does not exist"
Sep 11 01:17:18.787469 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.787444    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"]
Sep 11 01:17:18.790678 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:18.790654    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef3c90aa-57d456c986-xx76b"]
Sep 11 01:17:19.036995 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:19.036922    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c99998c-f382-40c4-8dc3-617349afcea4" path="/var/lib/kubelet/pods/1c99998c-f382-40c4-8dc3-617349afcea4/volumes"
Sep 11 01:17:37.205846 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:37.205807    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:17:42.899013 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.898983    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"]
Sep 11 01:17:42.899407 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.899393    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c99998c-f382-40c4-8dc3-617349afcea4" containerName="payload-pre-processing"
Sep 11 01:17:42.899407 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.899407    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c99998c-f382-40c4-8dc3-617349afcea4" containerName="payload-pre-processing"
Sep 11 01:17:42.899493 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.899470    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c99998c-f382-40c4-8dc3-617349afcea4" containerName="payload-pre-processing"
Sep 11 01:17:42.902614 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.902589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:42.911815 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.911792    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-76bb56-dockercfg-vqfnf\""
Sep 11 01:17:42.913546 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.913526    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-76bb56\""
Sep 11 01:17:42.914815 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.914793    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"]
Sep 11 01:17:42.917993 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.917975    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:42.944111 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.944084    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"]
Sep 11 01:17:42.998005 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.997969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cdab54-d747-4276-aced-bee307464f11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54\" (UID: \"37cdab54-d747-4276-aced-bee307464f11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:42.998183 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.998152    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09932b8-f03a-4618-a119-8b69c5b48569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk\" (UID: \"f09932b8-f03a-4618-a119-8b69c5b48569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:42.998350 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.998292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2mzq\" (UniqueName: \"kubernetes.io/projected/37cdab54-d747-4276-aced-bee307464f11-kube-api-access-q2mzq\") pod \"payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54\" (UID: \"37cdab54-d747-4276-aced-bee307464f11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:42.998498 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.998407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxp8\" (UniqueName: \"kubernetes.io/projected/f09932b8-f03a-4618-a119-8b69c5b48569-kube-api-access-wnxp8\") pod \"payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk\" (UID: \"f09932b8-f03a-4618-a119-8b69c5b48569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:42.999486 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:42.999461    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"]
Sep 11 01:17:43.099400 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.099369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09932b8-f03a-4618-a119-8b69c5b48569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk\" (UID: \"f09932b8-f03a-4618-a119-8b69c5b48569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:43.099587 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.099419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2mzq\" (UniqueName: \"kubernetes.io/projected/37cdab54-d747-4276-aced-bee307464f11-kube-api-access-q2mzq\") pod \"payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54\" (UID: \"37cdab54-d747-4276-aced-bee307464f11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:43.099587 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.099465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxp8\" (UniqueName: \"kubernetes.io/projected/f09932b8-f03a-4618-a119-8b69c5b48569-kube-api-access-wnxp8\") pod \"payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk\" (UID: \"f09932b8-f03a-4618-a119-8b69c5b48569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:43.099587 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.099486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cdab54-d747-4276-aced-bee307464f11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54\" (UID: \"37cdab54-d747-4276-aced-bee307464f11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:43.100132 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.100111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09932b8-f03a-4618-a119-8b69c5b48569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk\" (UID: \"f09932b8-f03a-4618-a119-8b69c5b48569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:43.100182 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.100144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cdab54-d747-4276-aced-bee307464f11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54\" (UID: \"37cdab54-d747-4276-aced-bee307464f11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:43.110621 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.110595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2mzq\" (UniqueName: \"kubernetes.io/projected/37cdab54-d747-4276-aced-bee307464f11-kube-api-access-q2mzq\") pod \"payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54\" (UID: \"37cdab54-d747-4276-aced-bee307464f11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:43.110743 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.110600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxp8\" (UniqueName: \"kubernetes.io/projected/f09932b8-f03a-4618-a119-8b69c5b48569-kube-api-access-wnxp8\") pod \"payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk\" (UID: \"f09932b8-f03a-4618-a119-8b69c5b48569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:43.212363 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.212279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:43.229778 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.229738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:43.491001 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.490979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"]
Sep 11 01:17:43.516712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.516688    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"]
Sep 11 01:17:43.522517 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:17:43.522488    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cdab54_d747_4276_aced_bee307464f11.slice/crio-181c89c0ce798637aded554ac67c83478a2071e29249bf7968580fad0220e395 WatchSource:0}: Error finding container 181c89c0ce798637aded554ac67c83478a2071e29249bf7968580fad0220e395: Status 404 returned error can't find the container with id 181c89c0ce798637aded554ac67c83478a2071e29249bf7968580fad0220e395
Sep 11 01:17:43.896353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.896292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54" event={"ID":"37cdab54-d747-4276-aced-bee307464f11","Type":"ContainerStarted","Data":"bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579"}
Sep 11 01:17:43.896574 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.896382    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:43.896574 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.896407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54" event={"ID":"37cdab54-d747-4276-aced-bee307464f11","Type":"ContainerStarted","Data":"181c89c0ce798637aded554ac67c83478a2071e29249bf7968580fad0220e395"}
Sep 11 01:17:43.898015 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.897980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk" event={"ID":"f09932b8-f03a-4618-a119-8b69c5b48569","Type":"ContainerStarted","Data":"8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501"}
Sep 11 01:17:43.898136 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.898039    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:43.898136 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.898062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk" event={"ID":"f09932b8-f03a-4618-a119-8b69c5b48569","Type":"ContainerStarted","Data":"adc03f9a92a203aa2fc1a645021e8f1d8000360b1a930556e1e72e91643745ee"}
Sep 11 01:17:43.915468 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.915421    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54" podStartSLOduration=1.915406417 podStartE2EDuration="1.915406417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:17:43.914734728 +0000 UTC m=+1193.451385929" watchObservedRunningTime="2026-09-11 01:17:43.915406417 +0000 UTC m=+1193.452057617"
Sep 11 01:17:43.931214 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:43.931176    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk" podStartSLOduration=1.931162976 podStartE2EDuration="1.931162976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:17:43.929935051 +0000 UTC m=+1193.466586253" watchObservedRunningTime="2026-09-11 01:17:43.931162976 +0000 UTC m=+1193.467814177"
Sep 11 01:17:48.275425 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.275385    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"]
Sep 11 01:17:48.275894 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.275631    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k" podUID="5624e6b3-30bf-4b3c-8f81-6b9eec75362b" containerName="maas-api" containerID="cri-o://2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76" gracePeriod=30
Sep 11 01:17:48.332181 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.332143    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"]
Sep 11 01:17:48.332462 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.332435    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z" podUID="1aee6c93-496b-4416-9b34-642d485819d4" containerName="payload-processing" containerID="cri-o://b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943" gracePeriod=30
Sep 11 01:17:48.590274 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.590244    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:17:48.608903 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.608877    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:17:48.753221 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753187    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqkz2\" (UniqueName: \"kubernetes.io/projected/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-kube-api-access-qqkz2\") pod \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") "
Sep 11 01:17:48.753439 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753245    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee6c93-496b-4416-9b34-642d485819d4-plugins-config-volume\") pod \"1aee6c93-496b-4416-9b34-642d485819d4\" (UID: \"1aee6c93-496b-4416-9b34-642d485819d4\") "
Sep 11 01:17:48.753439 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753276    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-metrics-certs\") pod \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") "
Sep 11 01:17:48.753439 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753298    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-maas-api-tls\") pod \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") "
Sep 11 01:17:48.753439 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753390    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-openshift-service-ca\") pod \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\" (UID: \"5624e6b3-30bf-4b3c-8f81-6b9eec75362b\") "
Sep 11 01:17:48.753439 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753411    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcjg8\" (UniqueName: \"kubernetes.io/projected/1aee6c93-496b-4416-9b34-642d485819d4-kube-api-access-wcjg8\") pod \"1aee6c93-496b-4416-9b34-642d485819d4\" (UID: \"1aee6c93-496b-4416-9b34-642d485819d4\") "
Sep 11 01:17:48.753720 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753661    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aee6c93-496b-4416-9b34-642d485819d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1aee6c93-496b-4416-9b34-642d485819d4" (UID: "1aee6c93-496b-4416-9b34-642d485819d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:17:48.753814 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753765    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5624e6b3-30bf-4b3c-8f81-6b9eec75362b" (UID: "5624e6b3-30bf-4b3c-8f81-6b9eec75362b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:17:48.753875 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.753837    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee6c93-496b-4416-9b34-642d485819d4-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:17:48.755609 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.755580    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-kube-api-access-qqkz2" (OuterVolumeSpecName: "kube-api-access-qqkz2") pod "5624e6b3-30bf-4b3c-8f81-6b9eec75362b" (UID: "5624e6b3-30bf-4b3c-8f81-6b9eec75362b"). InnerVolumeSpecName "kube-api-access-qqkz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:17:48.755702 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.755604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5624e6b3-30bf-4b3c-8f81-6b9eec75362b" (UID: "5624e6b3-30bf-4b3c-8f81-6b9eec75362b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:17:48.755702 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.755583    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5624e6b3-30bf-4b3c-8f81-6b9eec75362b" (UID: "5624e6b3-30bf-4b3c-8f81-6b9eec75362b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:17:48.755702 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.755662    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aee6c93-496b-4416-9b34-642d485819d4-kube-api-access-wcjg8" (OuterVolumeSpecName: "kube-api-access-wcjg8") pod "1aee6c93-496b-4416-9b34-642d485819d4" (UID: "1aee6c93-496b-4416-9b34-642d485819d4"). InnerVolumeSpecName "kube-api-access-wcjg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:17:48.854792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.854716    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-openshift-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:17:48.854792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.854748    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcjg8\" (UniqueName: \"kubernetes.io/projected/1aee6c93-496b-4416-9b34-642d485819d4-kube-api-access-wcjg8\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:17:48.854792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.854757    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqkz2\" (UniqueName: \"kubernetes.io/projected/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-kube-api-access-qqkz2\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:17:48.854792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.854767    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-metrics-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:17:48.854792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.854776    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5624e6b3-30bf-4b3c-8f81-6b9eec75362b-maas-api-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:17:48.927224 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.927194    2583 generic.go:356] "Generic (PLEG): container finished" podID="1aee6c93-496b-4416-9b34-642d485819d4" containerID="b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943" exitCode=0
Sep 11 01:17:48.927467 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.927242    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"
Sep 11 01:17:48.927467 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.927270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z" event={"ID":"1aee6c93-496b-4416-9b34-642d485819d4","Type":"ContainerDied","Data":"b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943"}
Sep 11 01:17:48.927467 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.927340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z" event={"ID":"1aee6c93-496b-4416-9b34-642d485819d4","Type":"ContainerDied","Data":"2e840cad4f0865bce128194b8795c790f155add26da7dda16d9e6628eb30acff"}
Sep 11 01:17:48.927467 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.927365    2583 scope.go:117] "RemoveContainer" containerID="b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943"
Sep 11 01:17:48.928822 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.928788    2583 generic.go:356] "Generic (PLEG): container finished" podID="5624e6b3-30bf-4b3c-8f81-6b9eec75362b" containerID="2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76" exitCode=0
Sep 11 01:17:48.928923 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.928820    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"
Sep 11 01:17:48.928923 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.928878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k" event={"ID":"5624e6b3-30bf-4b3c-8f81-6b9eec75362b","Type":"ContainerDied","Data":"2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76"}
Sep 11 01:17:48.928923 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.928916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k" event={"ID":"5624e6b3-30bf-4b3c-8f81-6b9eec75362b","Type":"ContainerDied","Data":"14c2472933763b1dd22b67a30d00cb9b08a5e7ea580f0a65e6d0643ffe7a857d"}
Sep 11 01:17:48.936643 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.936628    2583 scope.go:117] "RemoveContainer" containerID="b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943"
Sep 11 01:17:48.936915 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:17:48.936897    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943\": container with ID starting with b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943 not found: ID does not exist" containerID="b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943"
Sep 11 01:17:48.936986 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.936922    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943"} err="failed to get container status \"b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943\": rpc error: code = NotFound desc = could not find container \"b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943\": container with ID starting with b64bf625356e2804ec7af12d1f7ccf8ef5cb875c4d76733e0431aa61cdc19943 not found: ID does not exist"
Sep 11 01:17:48.936986 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.936937    2583 scope.go:117] "RemoveContainer" containerID="2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76"
Sep 11 01:17:48.945731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.945710    2583 scope.go:117] "RemoveContainer" containerID="2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76"
Sep 11 01:17:48.945979 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:17:48.945960    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76\": container with ID starting with 2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76 not found: ID does not exist" containerID="2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76"
Sep 11 01:17:48.946053 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.945984    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76"} err="failed to get container status \"2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76\": rpc error: code = NotFound desc = could not find container \"2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76\": container with ID starting with 2c55a0c1a727c826b103851ef51f3c64286fe5d5397c817855593e59e85bff76 not found: ID does not exist"
Sep 11 01:17:48.949101 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.949081    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"]
Sep 11 01:17:48.953434 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.953414    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-987d60-78c8b64f6f-hws9z"]
Sep 11 01:17:48.962826 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.962808    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"]
Sep 11 01:17:48.966560 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:48.966542    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-987d60-5cccb75994-xnd2k"]
Sep 11 01:17:49.040017 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:49.039988    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aee6c93-496b-4416-9b34-642d485819d4" path="/var/lib/kubelet/pods/1aee6c93-496b-4416-9b34-642d485819d4/volumes"
Sep 11 01:17:49.040449 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:49.040435    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5624e6b3-30bf-4b3c-8f81-6b9eec75362b" path="/var/lib/kubelet/pods/5624e6b3-30bf-4b3c-8f81-6b9eec75362b/volumes"
Sep 11 01:17:50.936088 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:50.936059    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:17:50.938076 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:50.938056    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:17:50.941861 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:50.941840    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:17:50.943333 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:50.943315    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:17:54.904508 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:54.904480    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:17:54.904941 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:54.904661    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:17:59.747689 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:17:59.747655    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"]
Sep 11 01:18:05.147904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.147870    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"]
Sep 11 01:18:05.148379 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.148134    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5" podUID="c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb" containerName="payload-processing" containerID="cri-o://27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6" gracePeriod=30
Sep 11 01:18:05.398058 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.398000    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:18:05.407532 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.407499    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-plugins-config-volume\") pod \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\" (UID: \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\") "
Sep 11 01:18:05.407652 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.407630    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77b7l\" (UniqueName: \"kubernetes.io/projected/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-kube-api-access-77b7l\") pod \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\" (UID: \"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb\") "
Sep 11 01:18:05.407898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.407874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb" (UID: "c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:18:05.407978 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.407958    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:05.409739 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.409715    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-kube-api-access-77b7l" (OuterVolumeSpecName: "kube-api-access-77b7l") pod "c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb" (UID: "c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb"). InnerVolumeSpecName "kube-api-access-77b7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:05.509429 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:05.509398    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77b7l\" (UniqueName: \"kubernetes.io/projected/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb-kube-api-access-77b7l\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:06.015724 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.015693    2583 generic.go:356] "Generic (PLEG): container finished" podID="c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb" containerID="27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6" exitCode=0
Sep 11 01:18:06.015895 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.015739    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"
Sep 11 01:18:06.015895 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.015750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5" event={"ID":"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb","Type":"ContainerDied","Data":"27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6"}
Sep 11 01:18:06.015895 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.015791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5" event={"ID":"c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb","Type":"ContainerDied","Data":"38543e1575b4d0798eb82e6425b139d5eba2326ac53b82847776a80e7d39e54a"}
Sep 11 01:18:06.015895 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.015811    2583 scope.go:117] "RemoveContainer" containerID="27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6"
Sep 11 01:18:06.026501 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.026482    2583 scope.go:117] "RemoveContainer" containerID="27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6"
Sep 11 01:18:06.026780 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:06.026757    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6\": container with ID starting with 27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6 not found: ID does not exist" containerID="27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6"
Sep 11 01:18:06.026825 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.026794    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6"} err="failed to get container status \"27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6\": rpc error: code = NotFound desc = could not find container \"27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6\": container with ID starting with 27c7a18d599cbd85996f38c6810da6c4205b26bf188c2efa6ce9c910e95fdbb6 not found: ID does not exist"
Sep 11 01:18:06.037766 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.037740    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"]
Sep 11 01:18:06.041187 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.041163    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cd176e-fb97d86c5-4z2c5"]
Sep 11 01:18:06.375425 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.375392    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:18:06.622327 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.621703    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"]
Sep 11 01:18:06.622327 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622237    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb" containerName="payload-processing"
Sep 11 01:18:06.622327 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622253    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb" containerName="payload-processing"
Sep 11 01:18:06.622327 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622269    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5624e6b3-30bf-4b3c-8f81-6b9eec75362b" containerName="maas-api"
Sep 11 01:18:06.622327 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622277    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5624e6b3-30bf-4b3c-8f81-6b9eec75362b" containerName="maas-api"
Sep 11 01:18:06.622327 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622317    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aee6c93-496b-4416-9b34-642d485819d4" containerName="payload-processing"
Sep 11 01:18:06.622327 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622326    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aee6c93-496b-4416-9b34-642d485819d4" containerName="payload-processing"
Sep 11 01:18:06.622787 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622418    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aee6c93-496b-4416-9b34-642d485819d4" containerName="payload-processing"
Sep 11 01:18:06.622787 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622428    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5624e6b3-30bf-4b3c-8f81-6b9eec75362b" containerName="maas-api"
Sep 11 01:18:06.622787 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.622438    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb" containerName="payload-processing"
Sep 11 01:18:06.632035 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.626994    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.632035 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.629419    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4f6824-openshift-default-dockercfg-tnct5\""
Sep 11 01:18:06.637158 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.636735    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"]
Sep 11 01:18:06.719147 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-credential-socket\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.719444 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-socket\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.719444 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-podinfo\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.719623 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-token\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.719688 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-envoy\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.719688 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-data\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.719789 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgl9t\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-kube-api-access-mgl9t\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.719789 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.719885 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.719874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-certs\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821003 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.820965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl9t\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-kube-api-access-mgl9t\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-certs\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-credential-socket\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-socket\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-podinfo\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821512 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-token\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821512 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-envoy\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821512 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-data\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821986 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-data\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.821986 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.821854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.822177 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.822019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-socket\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.822177 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.822105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-credential-socket\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.822549 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.822525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-certs\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.824072 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.824043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-envoy\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.824646 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.824627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-podinfo\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.828875 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.828848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-token\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.828970 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.828927    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgl9t\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-kube-api-access-mgl9t\") pod \"e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") " pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:06.941130 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:06.941059    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:07.037797 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:07.037766    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb" path="/var/lib/kubelet/pods/c30fe5b7-b2d1-495b-bbd3-81c1dacee2bb/volumes"
Sep 11 01:18:07.222911 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:07.222880    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"]
Sep 11 01:18:07.224518 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:18:07.224494    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf44596c_92c9_4b02_9db8_d2ee0b5b85af.slice/crio-9bead5267f563bf06635599cafd2a63d5f0892110363d808d2876d54e11ca728 WatchSource:0}: Error finding container 9bead5267f563bf06635599cafd2a63d5f0892110363d808d2876d54e11ca728: Status 404 returned error can't find the container with id 9bead5267f563bf06635599cafd2a63d5f0892110363d808d2876d54e11ca728
Sep 11 01:18:07.226917 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:07.226885    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:07.227013 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:07.226949    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:07.227013 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:07.226978    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:08.032357 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:08.032325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" event={"ID":"af44596c-92c9-4b02-9db8-d2ee0b5b85af","Type":"ContainerStarted","Data":"e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77"}
Sep 11 01:18:08.032847 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:08.032368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" event={"ID":"af44596c-92c9-4b02-9db8-d2ee0b5b85af","Type":"ContainerStarted","Data":"9bead5267f563bf06635599cafd2a63d5f0892110363d808d2876d54e11ca728"}
Sep 11 01:18:08.053566 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:08.053517    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" podStartSLOduration=2.0535029 podStartE2EDuration="2.0535029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:18:08.05115932 +0000 UTC m=+1217.587810521" watchObservedRunningTime="2026-09-11 01:18:08.0535029 +0000 UTC m=+1217.590154101"
Sep 11 01:18:08.942151 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:08.942120    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:08.947518 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:08.947493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:09.045390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:09.045353    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:09.046396 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:09.046375    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:11.218691 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.218661    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"]
Sep 11 01:18:11.223575 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.223551    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.225872 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.225849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4f6824-serving-cert\""
Sep 11 01:18:11.234608 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.234586    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"]
Sep 11 01:18:11.264618 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.264592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1642c6ef-7c39-41f8-86d4-d1f7780aca05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.264750 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.264675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.264750 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.264697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jkx\" (UniqueName: \"kubernetes.io/projected/1642c6ef-7c39-41f8-86d4-d1f7780aca05-kube-api-access-p6jkx\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.264750 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.264724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.365403 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.365372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1642c6ef-7c39-41f8-86d4-d1f7780aca05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.365541 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.365508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.365541 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.365536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jkx\" (UniqueName: \"kubernetes.io/projected/1642c6ef-7c39-41f8-86d4-d1f7780aca05-kube-api-access-p6jkx\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.365629 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.365564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.366077 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.366054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1642c6ef-7c39-41f8-86d4-d1f7780aca05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.368150 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.368125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.368253 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.368125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.373811 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.373786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jkx\" (UniqueName: \"kubernetes.io/projected/1642c6ef-7c39-41f8-86d4-d1f7780aca05-kube-api-access-p6jkx\") pod \"maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.536915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.536832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:11.818354 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:11.818330    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"]
Sep 11 01:18:11.820419 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:18:11.820391    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1642c6ef_7c39_41f8_86d4_d1f7780aca05.slice/crio-3cc6c17f7189d28400b8a28e973e3354869083607283e2a550c1f59ffb901ef9 WatchSource:0}: Error finding container 3cc6c17f7189d28400b8a28e973e3354869083607283e2a550c1f59ffb901ef9: Status 404 returned error can't find the container with id 3cc6c17f7189d28400b8a28e973e3354869083607283e2a550c1f59ffb901ef9
Sep 11 01:18:12.062083 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:12.062049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r" event={"ID":"1642c6ef-7c39-41f8-86d4-d1f7780aca05","Type":"ContainerStarted","Data":"3cc6c17f7189d28400b8a28e973e3354869083607283e2a550c1f59ffb901ef9"}
Sep 11 01:18:13.074365 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:13.074296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r" event={"ID":"1642c6ef-7c39-41f8-86d4-d1f7780aca05","Type":"ContainerStarted","Data":"93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e"}
Sep 11 01:18:13.074812 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:13.074411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:13.098729 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:13.098676    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r" podStartSLOduration=1.7260543510000002 podStartE2EDuration="2.098661445s" totalImagesPullingTime="372.607094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:18:13.096489421 +0000 UTC m=+1222.633140624" watchObservedRunningTime="2026-09-11 01:18:13.098661445 +0000 UTC m=+1222.635312679"
Sep 11 01:18:14.766545 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:14.766509    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6" podUID="8e5088ba-35be-425c-917f-4a3eb39094f9" containerName="main" containerID="cri-o://640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682" gracePeriod=45
Sep 11 01:18:15.031342 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.031264    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:18:15.086927 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.086897    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e5088ba-35be-425c-917f-4a3eb39094f9" containerID="640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682" exitCode=0
Sep 11 01:18:15.087086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.086945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6" event={"ID":"8e5088ba-35be-425c-917f-4a3eb39094f9","Type":"ContainerDied","Data":"640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682"}
Sep 11 01:18:15.087086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.086953    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"
Sep 11 01:18:15.087086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.086980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6" event={"ID":"8e5088ba-35be-425c-917f-4a3eb39094f9","Type":"ContainerDied","Data":"af6292c82b78b21a4fee8c3f775e9f76c21973f53380d57f69833cf13ad0566a"}
Sep 11 01:18:15.087086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.087000    2583 scope.go:117] "RemoveContainer" containerID="640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682"
Sep 11 01:18:15.097409 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.097390    2583 scope.go:117] "RemoveContainer" containerID="640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682"
Sep 11 01:18:15.097681 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:15.097662    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682\": container with ID starting with 640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682 not found: ID does not exist" containerID="640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682"
Sep 11 01:18:15.097750 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.097695    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682"} err="failed to get container status \"640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682\": rpc error: code = NotFound desc = could not find container \"640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682\": container with ID starting with 640b3eee6def291aa3257e3c8c6f69cf70e3fd2a188fa1ed074db6dc19e1f682 not found: ID does not exist"
Sep 11 01:18:15.102003 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.101984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-model-cache\") pod \"8e5088ba-35be-425c-917f-4a3eb39094f9\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") "
Sep 11 01:18:15.102086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-tmp-dir\") pod \"8e5088ba-35be-425c-917f-4a3eb39094f9\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") "
Sep 11 01:18:15.102086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-dshm\") pod \"8e5088ba-35be-425c-917f-4a3eb39094f9\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") "
Sep 11 01:18:15.102160 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5pjj\" (UniqueName: \"kubernetes.io/projected/8e5088ba-35be-425c-917f-4a3eb39094f9-kube-api-access-r5pjj\") pod \"8e5088ba-35be-425c-917f-4a3eb39094f9\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") "
Sep 11 01:18:15.102160 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102154    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5088ba-35be-425c-917f-4a3eb39094f9-tls-certs\") pod \"8e5088ba-35be-425c-917f-4a3eb39094f9\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") "
Sep 11 01:18:15.102258 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102194    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-home\") pod \"8e5088ba-35be-425c-917f-4a3eb39094f9\" (UID: \"8e5088ba-35be-425c-917f-4a3eb39094f9\") "
Sep 11 01:18:15.102258 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102215    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e5088ba-35be-425c-917f-4a3eb39094f9" (UID: "8e5088ba-35be-425c-917f-4a3eb39094f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:15.102385 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e5088ba-35be-425c-917f-4a3eb39094f9" (UID: "8e5088ba-35be-425c-917f-4a3eb39094f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:15.102733 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102519    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-home" (OuterVolumeSpecName: "home") pod "8e5088ba-35be-425c-917f-4a3eb39094f9" (UID: "8e5088ba-35be-425c-917f-4a3eb39094f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:15.102733 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102681    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-model-cache\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:15.102733 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102699    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-tmp-dir\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:15.102733 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.102711    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-home\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:15.104445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.104422    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5088ba-35be-425c-917f-4a3eb39094f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e5088ba-35be-425c-917f-4a3eb39094f9" (UID: "8e5088ba-35be-425c-917f-4a3eb39094f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:18:15.104684 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.104664    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-dshm" (OuterVolumeSpecName: "dshm") pod "8e5088ba-35be-425c-917f-4a3eb39094f9" (UID: "8e5088ba-35be-425c-917f-4a3eb39094f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:15.104788 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.104662    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5088ba-35be-425c-917f-4a3eb39094f9-kube-api-access-r5pjj" (OuterVolumeSpecName: "kube-api-access-r5pjj") pod "8e5088ba-35be-425c-917f-4a3eb39094f9" (UID: "8e5088ba-35be-425c-917f-4a3eb39094f9"). InnerVolumeSpecName "kube-api-access-r5pjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:15.203728 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.203701    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e5088ba-35be-425c-917f-4a3eb39094f9-dshm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:15.203728 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.203727    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5pjj\" (UniqueName: \"kubernetes.io/projected/8e5088ba-35be-425c-917f-4a3eb39094f9-kube-api-access-r5pjj\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:15.203925 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.203739    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5088ba-35be-425c-917f-4a3eb39094f9-tls-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:15.408027 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.407995    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"]
Sep 11 01:18:15.411667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:15.411645    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cd176e/premium-w4-cd176e-kserve-c5b8ddb6-7d7t6"]
Sep 11 01:18:17.037094 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:17.037063    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5088ba-35be-425c-917f-4a3eb39094f9" path="/var/lib/kubelet/pods/8e5088ba-35be-425c-917f-4a3eb39094f9/volumes"
Sep 11 01:18:19.085781 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:19.085756    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:18:29.175024 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:29.174987    2583 generic.go:356] "Generic (PLEG): container finished" podID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerID="e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77" exitCode=0
Sep 11 01:18:29.175440 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:29.175054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" event={"ID":"af44596c-92c9-4b02-9db8-d2ee0b5b85af","Type":"ContainerDied","Data":"e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77"}
Sep 11 01:18:29.175485 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:29.175447    2583 scope.go:117] "RemoveContainer" containerID="e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77"
Sep 11 01:18:29.176513 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:29.176489    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:29.176576 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:29.176556    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:29.176612 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:29.176589    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:30.182001 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:30.181964    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" event={"ID":"af44596c-92c9-4b02-9db8-d2ee0b5b85af","Type":"ContainerStarted","Data":"acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8"}
Sep 11 01:18:30.941737 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:30.941696    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:30.941936 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:30.941823    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 01:18:30.941936 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:30.941867    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 01:18:31.061686 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:31.061658    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"]
Sep 11 01:18:31.063260 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:31.062016    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy" containerID="cri-o://ad59b38908005c1bf4e53786f036af4c391b1c1a56a9e4f3c119dd9435f01bdd" gracePeriod=30
Sep 11 01:18:31.945331 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:31.944706    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 01:18:31.945331 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:31.944776    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 01:18:32.942286 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:32.942226    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 01:18:32.942599 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:32.942571    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 01:18:33.093871 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:33.093840    2583 patch_prober.go:28] interesting pod/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 01:18:33.094225 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:33.093895    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 01:18:33.941950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:33.941911    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 01:18:33.942228 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:33.942198    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 01:18:34.942365 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:34.942319    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 01:18:34.942848 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:34.942392    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 01:18:35.945642 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:35.945612    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:35.946052 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:35.945989    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:35.946792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:35.946771    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:18:36.216154 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.216062    2583 generic.go:356] "Generic (PLEG): container finished" podID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerID="ad59b38908005c1bf4e53786f036af4c391b1c1a56a9e4f3c119dd9435f01bdd" exitCode=0
Sep 11 01:18:36.216154 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.216132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" event={"ID":"75a7c7ac-4e1c-440b-8864-d7a67160d8d9","Type":"ContainerDied","Data":"ad59b38908005c1bf4e53786f036af4c391b1c1a56a9e4f3c119dd9435f01bdd"}
Sep 11 01:18:36.216390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.216194    2583 scope.go:117] "RemoveContainer" containerID="7bd806b565b698df7c8ce35cf0f61d853381adc5a10a8363ce292d35ccc43beb"
Sep 11 01:18:36.315683 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.315655    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:18:36.414256 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414220    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-certs\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414464 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414292    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-token\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414464 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414349    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-envoy\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414464 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414385    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-data\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414464 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414419    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-credential-socket\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414464 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414448    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-podinfo\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414550    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istiod-ca-cert\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414571    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:36.414731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414602    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-socket\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414641    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6mrw\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-kube-api-access-x6mrw\") pod \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\" (UID: \"75a7c7ac-4e1c-440b-8864-d7a67160d8d9\") "
Sep 11 01:18:36.414731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414703    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:36.414731 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414718    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:36.415019 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.414972    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:36.415019 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.415007    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:36.415126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.415028    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:36.415126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.415046    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:36.415126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.415087    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:18:36.416914 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.416881    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-kube-api-access-x6mrw" (OuterVolumeSpecName: "kube-api-access-x6mrw") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "kube-api-access-x6mrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:36.416914 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.416881    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:18:36.417054 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.416974    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:18:36.417054 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.416980    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "75a7c7ac-4e1c-440b-8864-d7a67160d8d9" (UID: "75a7c7ac-4e1c-440b-8864-d7a67160d8d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:36.515995 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.515910    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:36.515995 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.515938    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:36.515995 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.515951    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6mrw\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-kube-api-access-x6mrw\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:36.515995 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.515964    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:36.515995 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.515975    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:36.515995 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:36.515986    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a7c7ac-4e1c-440b-8864-d7a67160d8d9-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:37.224527 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.224493    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"
Sep 11 01:18:37.225020 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.224540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n" event={"ID":"75a7c7ac-4e1c-440b-8864-d7a67160d8d9","Type":"ContainerDied","Data":"c8922ded18840437c1828256420536abf061d26d256d64273dde9b2059f7e6bf"}
Sep 11 01:18:37.225020 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.224608    2583 scope.go:117] "RemoveContainer" containerID="ad59b38908005c1bf4e53786f036af4c391b1c1a56a9e4f3c119dd9435f01bdd"
Sep 11 01:18:37.241452 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.241430    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"]
Sep 11 01:18:37.245267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.245241    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-987d60-openshift-default-6768b96b4d-qns7n"]
Sep 11 01:18:37.672816 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.672784    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7962-04ac6c"]
Sep 11 01:18:37.673157 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673145    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy"
Sep 11 01:18:37.673199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673158    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy"
Sep 11 01:18:37.673199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673189    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy"
Sep 11 01:18:37.673199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673195    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy"
Sep 11 01:18:37.673289 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673203    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5088ba-35be-425c-917f-4a3eb39094f9" containerName="main"
Sep 11 01:18:37.673289 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673209    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5088ba-35be-425c-917f-4a3eb39094f9" containerName="main"
Sep 11 01:18:37.673289 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673272    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy"
Sep 11 01:18:37.673289 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673280    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" containerName="istio-proxy"
Sep 11 01:18:37.673289 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.673289    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5088ba-35be-425c-917f-4a3eb39094f9" containerName="main"
Sep 11 01:18:37.675799 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.675784    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:37.678119 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.678077    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7gdtv\""
Sep 11 01:18:37.679909 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.679884    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7962-04ac6c"]
Sep 11 01:18:37.827273 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.827240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxv4w\" (UniqueName: \"kubernetes.io/projected/adda1b6c-3831-4ce7-be06-6a21eb2dd7a1-kube-api-access-zxv4w\") pod \"test-curl-7962-04ac6c\" (UID: \"adda1b6c-3831-4ce7-be06-6a21eb2dd7a1\") " pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:37.928719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.928629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxv4w\" (UniqueName: \"kubernetes.io/projected/adda1b6c-3831-4ce7-be06-6a21eb2dd7a1-kube-api-access-zxv4w\") pod \"test-curl-7962-04ac6c\" (UID: \"adda1b6c-3831-4ce7-be06-6a21eb2dd7a1\") " pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:37.936099 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.936075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxv4w\" (UniqueName: \"kubernetes.io/projected/adda1b6c-3831-4ce7-be06-6a21eb2dd7a1-kube-api-access-zxv4w\") pod \"test-curl-7962-04ac6c\" (UID: \"adda1b6c-3831-4ce7-be06-6a21eb2dd7a1\") " pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:37.986247 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:37.986221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:38.090769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:38.090730    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:18:38.256036 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:38.256008    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7962-04ac6c"]
Sep 11 01:18:38.258133 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:18:38.258099    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadda1b6c_3831_4ce7_be06_6a21eb2dd7a1.slice/crio-32099dea0fb7f8a7b1c9e2972ee13ced5c6645eb7a9468b2a161d79558a88c61 WatchSource:0}: Error finding container 32099dea0fb7f8a7b1c9e2972ee13ced5c6645eb7a9468b2a161d79558a88c61: Status 404 returned error can't find the container with id 32099dea0fb7f8a7b1c9e2972ee13ced5c6645eb7a9468b2a161d79558a88c61
Sep 11 01:18:39.037977 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:39.037947    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a7c7ac-4e1c-440b-8864-d7a67160d8d9" path="/var/lib/kubelet/pods/75a7c7ac-4e1c-440b-8864-d7a67160d8d9/volumes"
Sep 11 01:18:39.239627 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:39.239591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7962-04ac6c" event={"ID":"adda1b6c-3831-4ce7-be06-6a21eb2dd7a1","Type":"ContainerStarted","Data":"e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4"}
Sep 11 01:18:39.239826 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:39.239640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7962-04ac6c" event={"ID":"adda1b6c-3831-4ce7-be06-6a21eb2dd7a1","Type":"ContainerStarted","Data":"32099dea0fb7f8a7b1c9e2972ee13ced5c6645eb7a9468b2a161d79558a88c61"}
Sep 11 01:18:39.254565 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:39.254482    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7962-04ac6c" podStartSLOduration=1.515335516 podStartE2EDuration="2.254467871s" totalImagesPullingTime="739.132355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:18:39.252707701 +0000 UTC m=+1248.789358903" watchObservedRunningTime="2026-09-11 01:18:39.254467871 +0000 UTC m=+1248.791119071"
Sep 11 01:18:39.892873 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:39.892790    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7962-04ac6c"]
Sep 11 01:18:39.895384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:39.895355    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7962-04ac6c"]
Sep 11 01:18:40.016799 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.016707    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7962-5ee647"]
Sep 11 01:18:40.017198 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.017179    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" containerName="test-curl-7962-04ac6c"
Sep 11 01:18:40.017286 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.017200    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" containerName="test-curl-7962-04ac6c"
Sep 11 01:18:40.017389 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.017360    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" containerName="test-curl-7962-04ac6c"
Sep 11 01:18:40.020418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.020397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-5ee647"
Sep 11 01:18:40.025963 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.025939    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7962-5ee647"]
Sep 11 01:18:40.149767 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.149720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzsd\" (UniqueName: \"kubernetes.io/projected/10670278-48a7-4dac-a878-1508b63ceb01-kube-api-access-qfzsd\") pod \"test-curl-7962-5ee647\" (UID: \"10670278-48a7-4dac-a878-1508b63ceb01\") " pod="opendatahub/test-curl-7962-5ee647"
Sep 11 01:18:40.245301 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:40.245263    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7962-04ac6c" container="test-curl-7962-04ac6c"
Sep 11 01:18:40.250507 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.250483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzsd\" (UniqueName: \"kubernetes.io/projected/10670278-48a7-4dac-a878-1508b63ceb01-kube-api-access-qfzsd\") pod \"test-curl-7962-5ee647\" (UID: \"10670278-48a7-4dac-a878-1508b63ceb01\") " pod="opendatahub/test-curl-7962-5ee647"
Sep 11 01:18:40.258760 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.258736    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzsd\" (UniqueName: \"kubernetes.io/projected/10670278-48a7-4dac-a878-1508b63ceb01-kube-api-access-qfzsd\") pod \"test-curl-7962-5ee647\" (UID: \"10670278-48a7-4dac-a878-1508b63ceb01\") " pod="opendatahub/test-curl-7962-5ee647"
Sep 11 01:18:40.333942 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.333914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-5ee647"
Sep 11 01:18:40.600190 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:40.600167    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7962-5ee647"]
Sep 11 01:18:40.602067 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:18:40.602041    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10670278_48a7_4dac_a878_1508b63ceb01.slice/crio-447633b7c580ae11c2e706918268c5a0f1f98e2142aa615f6f63b6b84c0cc286 WatchSource:0}: Error finding container 447633b7c580ae11c2e706918268c5a0f1f98e2142aa615f6f63b6b84c0cc286: Status 404 returned error can't find the container with id 447633b7c580ae11c2e706918268c5a0f1f98e2142aa615f6f63b6b84c0cc286
Sep 11 01:18:41.252126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:41.252089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7962-5ee647" event={"ID":"10670278-48a7-4dac-a878-1508b63ceb01","Type":"ContainerStarted","Data":"89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471"}
Sep 11 01:18:41.252627 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:41.252123    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7962-04ac6c" podUID="adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" containerName="test-curl-7962-04ac6c" containerID="cri-o://e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4" gracePeriod=2
Sep 11 01:18:41.252627 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:41.252132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7962-5ee647" event={"ID":"10670278-48a7-4dac-a878-1508b63ceb01","Type":"ContainerStarted","Data":"447633b7c580ae11c2e706918268c5a0f1f98e2142aa615f6f63b6b84c0cc286"}
Sep 11 01:18:41.266514 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:41.266470    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7962-5ee647" podStartSLOduration=1.26645576 podStartE2EDuration="1.26645576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:18:41.264937652 +0000 UTC m=+1250.801588853" watchObservedRunningTime="2026-09-11 01:18:41.26645576 +0000 UTC m=+1250.803106989"
Sep 11 01:18:41.868187 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:41.868152    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7962-5ee647"]
Sep 11 01:18:41.871327 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:41.871278    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7962-5ee647"]
Sep 11 01:18:42.013360 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.013259    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7962-5de62b"]
Sep 11 01:18:42.013656 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.013642    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10670278-48a7-4dac-a878-1508b63ceb01" containerName="test-curl-7962-5ee647"
Sep 11 01:18:42.013703 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.013657    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="10670278-48a7-4dac-a878-1508b63ceb01" containerName="test-curl-7962-5ee647"
Sep 11 01:18:42.013754 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.013732    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="10670278-48a7-4dac-a878-1508b63ceb01" containerName="test-curl-7962-5ee647"
Sep 11 01:18:42.015586 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.015571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-5de62b"
Sep 11 01:18:42.022666 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.022609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7962-5de62b"]
Sep 11 01:18:42.069873 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.069841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgllf\" (UniqueName: \"kubernetes.io/projected/6e8916d2-b210-490d-a930-a2e08e1369e0-kube-api-access-lgllf\") pod \"test-curl-7962-5de62b\" (UID: \"6e8916d2-b210-490d-a930-a2e08e1369e0\") " pod="opendatahub/test-curl-7962-5de62b"
Sep 11 01:18:42.170399 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.170358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgllf\" (UniqueName: \"kubernetes.io/projected/6e8916d2-b210-490d-a930-a2e08e1369e0-kube-api-access-lgllf\") pod \"test-curl-7962-5de62b\" (UID: \"6e8916d2-b210-490d-a930-a2e08e1369e0\") " pod="opendatahub/test-curl-7962-5de62b"
Sep 11 01:18:42.177942 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.177918    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgllf\" (UniqueName: \"kubernetes.io/projected/6e8916d2-b210-490d-a930-a2e08e1369e0-kube-api-access-lgllf\") pod \"test-curl-7962-5de62b\" (UID: \"6e8916d2-b210-490d-a930-a2e08e1369e0\") " pod="opendatahub/test-curl-7962-5de62b"
Sep 11 01:18:42.257363 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:42.257331    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7962-5ee647" container="test-curl-7962-5ee647"
Sep 11 01:18:42.327738 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.327708    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-5de62b"
Sep 11 01:18:42.594219 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:42.594180    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7962-5de62b"]
Sep 11 01:18:42.596265 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:18:42.596238    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e8916d2_b210_490d_a930_a2e08e1369e0.slice/crio-8a520e622d591e6597ecb17b0858dcdb5d1a5f286a812073f51ff5879d7c98ce WatchSource:0}: Error finding container 8a520e622d591e6597ecb17b0858dcdb5d1a5f286a812073f51ff5879d7c98ce: Status 404 returned error can't find the container with id 8a520e622d591e6597ecb17b0858dcdb5d1a5f286a812073f51ff5879d7c98ce
Sep 11 01:18:43.264292 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.264254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7962-5de62b" event={"ID":"6e8916d2-b210-490d-a930-a2e08e1369e0","Type":"ContainerStarted","Data":"a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e"}
Sep 11 01:18:43.264687 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.264315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7962-5de62b" event={"ID":"6e8916d2-b210-490d-a930-a2e08e1369e0","Type":"ContainerStarted","Data":"8a520e622d591e6597ecb17b0858dcdb5d1a5f286a812073f51ff5879d7c98ce"}
Sep 11 01:18:43.264687 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.264351    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7962-5ee647" podUID="10670278-48a7-4dac-a878-1508b63ceb01" containerName="test-curl-7962-5ee647" containerID="cri-o://89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471" gracePeriod=2
Sep 11 01:18:43.281316 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.281270    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7962-5de62b" podStartSLOduration=1.281257115 podStartE2EDuration="1.281257115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:18:43.279299602 +0000 UTC m=+1252.815950802" watchObservedRunningTime="2026-09-11 01:18:43.281257115 +0000 UTC m=+1252.817908315"
Sep 11 01:18:43.435195 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.435171    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:43.478272 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.478249    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxv4w\" (UniqueName: \"kubernetes.io/projected/adda1b6c-3831-4ce7-be06-6a21eb2dd7a1-kube-api-access-zxv4w\") pod \"adda1b6c-3831-4ce7-be06-6a21eb2dd7a1\" (UID: \"adda1b6c-3831-4ce7-be06-6a21eb2dd7a1\") "
Sep 11 01:18:43.480478 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.480450    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adda1b6c-3831-4ce7-be06-6a21eb2dd7a1-kube-api-access-zxv4w" (OuterVolumeSpecName: "kube-api-access-zxv4w") pod "adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" (UID: "adda1b6c-3831-4ce7-be06-6a21eb2dd7a1"). InnerVolumeSpecName "kube-api-access-zxv4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:43.579241 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.579211    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxv4w\" (UniqueName: \"kubernetes.io/projected/adda1b6c-3831-4ce7-be06-6a21eb2dd7a1-kube-api-access-zxv4w\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:43.870482 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.870409    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7962-5de62b"]
Sep 11 01:18:43.876273 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:43.876250    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7962-5de62b"]
Sep 11 01:18:44.270348 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.270244    2583 generic.go:356] "Generic (PLEG): container finished" podID="adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" containerID="e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4" exitCode=137
Sep 11 01:18:44.270348 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.270287    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:44.270817 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.270361    2583 scope.go:117] "RemoveContainer" containerID="e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4"
Sep 11 01:18:44.270817 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:44.270628    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7962-5de62b" container="test-curl-7962-5de62b"
Sep 11 01:18:44.279666 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.279649    2583 scope.go:117] "RemoveContainer" containerID="e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4"
Sep 11 01:18:44.279937 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:44.279919    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4\": container with ID starting with e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4 not found: ID does not exist" containerID="e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4"
Sep 11 01:18:44.279999 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.279949    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4"} err="failed to get container status \"e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4\": rpc error: code = NotFound desc = could not find container \"e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4\": container with ID starting with e0517c85595e3db777bebd2d23810d0e5c4beb7854a0c1d06cae9cba0acb07e4 not found: ID does not exist"
Sep 11 01:18:44.280683 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:44.280662    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7962-04ac6c\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:44.768972 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.768942    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"]
Sep 11 01:18:44.769445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.769427    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8916d2-b210-490d-a930-a2e08e1369e0" containerName="test-curl-7962-5de62b"
Sep 11 01:18:44.769445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.769447    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8916d2-b210-490d-a930-a2e08e1369e0" containerName="test-curl-7962-5de62b"
Sep 11 01:18:44.769638 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.769548    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8916d2-b210-490d-a930-a2e08e1369e0" containerName="test-curl-7962-5de62b"
Sep 11 01:18:44.772264 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.772249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.774364 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:44.774336    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7962-04ac6c\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" pod="opendatahub/test-curl-7962-04ac6c"
Sep 11 01:18:44.774497 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.774481    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6f69f6e6-openshift-default-dockercfg-zxlk6\""
Sep 11 01:18:44.782577 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.782558    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"]
Sep 11 01:18:44.789115 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-socket\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.789232 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789127    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-certs\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.789232 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-envoy\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.789232 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-data\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.789446 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-podinfo\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.789446 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-credential-socket\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.789446 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-token\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.789446 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23685a78-f9fb-4132-bb71-7980c8d81f1f-istiod-ca-cert\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.789446 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.789350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgnlg\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-kube-api-access-tgnlg\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890031 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.889994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-socket\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-certs\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-envoy\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-data\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890358 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-podinfo\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890358 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-credential-socket\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890358 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-token\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890358 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23685a78-f9fb-4132-bb71-7980c8d81f1f-istiod-ca-cert\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890358 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnlg\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-kube-api-access-tgnlg\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890604 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-socket\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890604 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-certs\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890737 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-data\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.890791 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.890762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-credential-socket\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.891204 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.891182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23685a78-f9fb-4132-bb71-7980c8d81f1f-istiod-ca-cert\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.892523 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.892502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-envoy\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.892891 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.892866    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-podinfo\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.904554 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.904533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-token\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:44.915200 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:44.915177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnlg\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-kube-api-access-tgnlg\") pod \"e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") " pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:45.037870 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.037790    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adda1b6c-3831-4ce7-be06-6a21eb2dd7a1" path="/var/lib/kubelet/pods/adda1b6c-3831-4ce7-be06-6a21eb2dd7a1/volumes"
Sep 11 01:18:45.082861 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.082836    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:45.276119 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.276086    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7962-5de62b" podUID="6e8916d2-b210-490d-a930-a2e08e1369e0" containerName="test-curl-7962-5de62b" containerID="cri-o://a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e" gracePeriod=2
Sep 11 01:18:45.356875 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.356843    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"]
Sep 11 01:18:45.358363 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:18:45.358335    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23685a78_f9fb_4132_bb71_7980c8d81f1f.slice/crio-e3c856827139e3a420940bec06d39e1acf5e60a3878bde4404ffc68bd43944ea WatchSource:0}: Error finding container e3c856827139e3a420940bec06d39e1acf5e60a3878bde4404ffc68bd43944ea: Status 404 returned error can't find the container with id e3c856827139e3a420940bec06d39e1acf5e60a3878bde4404ffc68bd43944ea
Sep 11 01:18:45.360699 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.360669    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:45.360787 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.360730    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:45.360787 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.360757    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:18:45.446654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.446635    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-5ee647"
Sep 11 01:18:45.496398 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.496367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfzsd\" (UniqueName: \"kubernetes.io/projected/10670278-48a7-4dac-a878-1508b63ceb01-kube-api-access-qfzsd\") pod \"10670278-48a7-4dac-a878-1508b63ceb01\" (UID: \"10670278-48a7-4dac-a878-1508b63ceb01\") "
Sep 11 01:18:45.498346 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.498301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10670278-48a7-4dac-a878-1508b63ceb01-kube-api-access-qfzsd" (OuterVolumeSpecName: "kube-api-access-qfzsd") pod "10670278-48a7-4dac-a878-1508b63ceb01" (UID: "10670278-48a7-4dac-a878-1508b63ceb01"). InnerVolumeSpecName "kube-api-access-qfzsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:45.597426 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:45.597335    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzsd\" (UniqueName: \"kubernetes.io/projected/10670278-48a7-4dac-a878-1508b63ceb01-kube-api-access-qfzsd\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:46.282616 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:46.282536    2583 generic.go:356] "Generic (PLEG): container finished" podID="10670278-48a7-4dac-a878-1508b63ceb01" containerID="89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471" exitCode=137
Sep 11 01:18:46.282616 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:46.282586    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-5ee647"
Sep 11 01:18:46.283067 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:46.282630    2583 scope.go:117] "RemoveContainer" containerID="89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471"
Sep 11 01:18:46.284665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:46.284629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" event={"ID":"23685a78-f9fb-4132-bb71-7980c8d81f1f","Type":"ContainerStarted","Data":"6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05"}
Sep 11 01:18:46.284777 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:46.284696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" event={"ID":"23685a78-f9fb-4132-bb71-7980c8d81f1f","Type":"ContainerStarted","Data":"e3c856827139e3a420940bec06d39e1acf5e60a3878bde4404ffc68bd43944ea"}
Sep 11 01:18:46.292411 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:46.292396    2583 scope.go:117] "RemoveContainer" containerID="89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471"
Sep 11 01:18:46.292736 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:46.292719    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471\": container with ID starting with 89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471 not found: ID does not exist" containerID="89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471"
Sep 11 01:18:46.292775 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:46.292746    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471"} err="failed to get container status \"89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471\": rpc error: code = NotFound desc = could not find container \"89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471\": container with ID starting with 89a8e53a7b17fdd3dd7e89f35ad7a19cf06faef1ead292ace88a881c713d1471 not found: ID does not exist"
Sep 11 01:18:46.306431 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:46.306405    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7962-5ee647\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="10670278-48a7-4dac-a878-1508b63ceb01" pod="opendatahub/test-curl-7962-5ee647"
Sep 11 01:18:46.306976 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:46.306942    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" podStartSLOduration=2.306929614 podStartE2EDuration="2.306929614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:18:46.304552032 +0000 UTC m=+1255.841203232" watchObservedRunningTime="2026-09-11 01:18:46.306929614 +0000 UTC m=+1255.843580881"
Sep 11 01:18:47.038166 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.038108    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10670278-48a7-4dac-a878-1508b63ceb01" path="/var/lib/kubelet/pods/10670278-48a7-4dac-a878-1508b63ceb01/volumes"
Sep 11 01:18:47.083461 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.083433    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:47.088633 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.088609    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:47.295193 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.295164    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:47.296358 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.296337    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:18:47.459221 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.459201    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-5de62b"
Sep 11 01:18:47.516115 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.516096    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgllf\" (UniqueName: \"kubernetes.io/projected/6e8916d2-b210-490d-a930-a2e08e1369e0-kube-api-access-lgllf\") pod \"6e8916d2-b210-490d-a930-a2e08e1369e0\" (UID: \"6e8916d2-b210-490d-a930-a2e08e1369e0\") "
Sep 11 01:18:47.518128 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.518107    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8916d2-b210-490d-a930-a2e08e1369e0-kube-api-access-lgllf" (OuterVolumeSpecName: "kube-api-access-lgllf") pod "6e8916d2-b210-490d-a930-a2e08e1369e0" (UID: "6e8916d2-b210-490d-a930-a2e08e1369e0"). InnerVolumeSpecName "kube-api-access-lgllf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:18:47.617240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:47.617194    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgllf\" (UniqueName: \"kubernetes.io/projected/6e8916d2-b210-490d-a930-a2e08e1369e0-kube-api-access-lgllf\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:18:48.301232 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:48.301195    2583 generic.go:356] "Generic (PLEG): container finished" podID="6e8916d2-b210-490d-a930-a2e08e1369e0" containerID="a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e" exitCode=137
Sep 11 01:18:48.301775 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:48.301239    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7962-5de62b"
Sep 11 01:18:48.301775 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:48.301267    2583 scope.go:117] "RemoveContainer" containerID="a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e"
Sep 11 01:18:48.310330 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:48.310284    2583 scope.go:117] "RemoveContainer" containerID="a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e"
Sep 11 01:18:48.310637 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:48.310611    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e\": container with ID starting with a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e not found: ID does not exist" containerID="a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e"
Sep 11 01:18:48.310725 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:48.310645    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e"} err="failed to get container status \"a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e\": rpc error: code = NotFound desc = could not find container \"a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e\": container with ID starting with a21369bcfe50813b54a382462f58790cea029e23139325c32c2b72839902e48e not found: ID does not exist"
Sep 11 01:18:48.311676 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:18:48.311646    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7962-5de62b\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="6e8916d2-b210-490d-a930-a2e08e1369e0" pod="opendatahub/test-curl-7962-5de62b"
Sep 11 01:18:49.037800 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:49.037766    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e8916d2-b210-490d-a930-a2e08e1369e0" path="/var/lib/kubelet/pods/6e8916d2-b210-490d-a930-a2e08e1369e0/volumes"
Sep 11 01:18:50.250092 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.250061    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"]
Sep 11 01:18:50.253133 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.253117    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:50.255643 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.255621    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6f69f6e6\""
Sep 11 01:18:50.255769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.255625    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6f69f6e6-dockercfg-rcb65\""
Sep 11 01:18:50.262072 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.262050    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"]
Sep 11 01:18:50.347492 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.347459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4681e69-f131-4726-aaf6-afc46024335b-plugins-config-volume\") pod \"payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv\" (UID: \"e4681e69-f131-4726-aaf6-afc46024335b\") " pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:50.347669 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.347648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6frld\" (UniqueName: \"kubernetes.io/projected/e4681e69-f131-4726-aaf6-afc46024335b-kube-api-access-6frld\") pod \"payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv\" (UID: \"e4681e69-f131-4726-aaf6-afc46024335b\") " pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:50.448991 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.448954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6frld\" (UniqueName: \"kubernetes.io/projected/e4681e69-f131-4726-aaf6-afc46024335b-kube-api-access-6frld\") pod \"payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv\" (UID: \"e4681e69-f131-4726-aaf6-afc46024335b\") " pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:50.449178 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.449008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4681e69-f131-4726-aaf6-afc46024335b-plugins-config-volume\") pod \"payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv\" (UID: \"e4681e69-f131-4726-aaf6-afc46024335b\") " pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:50.449774 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.449755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4681e69-f131-4726-aaf6-afc46024335b-plugins-config-volume\") pod \"payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv\" (UID: \"e4681e69-f131-4726-aaf6-afc46024335b\") " pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:50.456789 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.456768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6frld\" (UniqueName: \"kubernetes.io/projected/e4681e69-f131-4726-aaf6-afc46024335b-kube-api-access-6frld\") pod \"payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv\" (UID: \"e4681e69-f131-4726-aaf6-afc46024335b\") " pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:50.564162 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.564136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:50.834436 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:50.834412    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"]
Sep 11 01:18:50.836095 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:18:50.836068    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4681e69_f131_4726_aaf6_afc46024335b.slice/crio-6c497b427e13d341ef82fc8687cace1a3551c2298a2c3e9ec8757c22fb32e80c WatchSource:0}: Error finding container 6c497b427e13d341ef82fc8687cace1a3551c2298a2c3e9ec8757c22fb32e80c: Status 404 returned error can't find the container with id 6c497b427e13d341ef82fc8687cace1a3551c2298a2c3e9ec8757c22fb32e80c
Sep 11 01:18:51.320090 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:51.320050    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv" event={"ID":"e4681e69-f131-4726-aaf6-afc46024335b","Type":"ContainerStarted","Data":"36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710"}
Sep 11 01:18:51.320570 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:51.320110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv" event={"ID":"e4681e69-f131-4726-aaf6-afc46024335b","Type":"ContainerStarted","Data":"6c497b427e13d341ef82fc8687cace1a3551c2298a2c3e9ec8757c22fb32e80c"}
Sep 11 01:18:51.320570 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:51.320155    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:18:51.347735 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:18:51.347658    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv" podStartSLOduration=1.347642836 podStartE2EDuration="1.347642836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:18:51.346743795 +0000 UTC m=+1260.883394998" watchObservedRunningTime="2026-09-11 01:18:51.347642836 +0000 UTC m=+1260.884294037"
Sep 11 01:19:02.329318 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:02.329266    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:19:04.184538 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:04.184501    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:19:15.445746 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:15.445711    2583 generic.go:356] "Generic (PLEG): container finished" podID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerID="6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05" exitCode=0
Sep 11 01:19:15.446146 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:15.445786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" event={"ID":"23685a78-f9fb-4132-bb71-7980c8d81f1f","Type":"ContainerDied","Data":"6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05"}
Sep 11 01:19:15.446260 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:15.446241    2583 scope.go:117] "RemoveContainer" containerID="6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05"
Sep 11 01:19:15.447418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:15.447383    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:19:15.447483 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:15.447461    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:19:15.447519 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:15.447492    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:19:16.454462 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:16.454429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" event={"ID":"23685a78-f9fb-4132-bb71-7980c8d81f1f","Type":"ContainerStarted","Data":"bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace"}
Sep 11 01:19:17.083706 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:17.083672    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:19:17.084992 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:17.084947    2583 patch_prober.go:28] interesting pod/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 01:19:17.085062 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:17.085018    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 01:19:17.295059 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:17.295013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:19:18.083993 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:18.083954    2583 patch_prober.go:28] interesting pod/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 01:19:18.084498 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:18.084011    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 01:19:19.083863 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:19.083814    2583 patch_prober.go:28] interesting pod/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 01:19:19.084048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:19.083880    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 01:19:20.084249 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:20.084212    2583 patch_prober.go:28] interesting pod/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 01:19:20.084673 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:20.084271    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 01:19:21.083383 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:21.083346    2583 patch_prober.go:28] interesting pod/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 01:19:21.083588 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:21.083405    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 01:19:21.977476 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:21.977443    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"]
Sep 11 01:19:21.980827 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:21.980803    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:21.983156 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:21.983139    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f4de10\"/\"distinct-w0-f4de10-kserve-self-signed-certs\""
Sep 11 01:19:21.992228 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:21.992206    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"]
Sep 11 01:19:22.057897 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.057866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-tmp-dir\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.058061 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.057910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-home\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.058061 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.058036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-model-cache\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.058147 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.058081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw558\" (UniqueName: \"kubernetes.io/projected/1ae37207-ef77-44de-9811-b131dc58f8e0-kube-api-access-bw558\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.058147 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.058112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-dshm\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.058233 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.058162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae37207-ef77-44de-9811-b131dc58f8e0-tls-certs\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.087707 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.087684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:19:22.088644 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.088627    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:19:22.160048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.160012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-tmp-dir\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.160724 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.160699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-home\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.161010 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.160989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-model-cache\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.161141 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.161125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw558\" (UniqueName: \"kubernetes.io/projected/1ae37207-ef77-44de-9811-b131dc58f8e0-kube-api-access-bw558\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.161258 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.161242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-dshm\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.161462 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.161443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae37207-ef77-44de-9811-b131dc58f8e0-tls-certs\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.162322 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.162284    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-model-cache\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.163429 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.162882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-home\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.163564 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.160637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-tmp-dir\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.164799 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.164776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae37207-ef77-44de-9811-b131dc58f8e0-tls-certs\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.165139 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.165118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-dshm\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.171469 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.171444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw558\" (UniqueName: \"kubernetes.io/projected/1ae37207-ef77-44de-9811-b131dc58f8e0-kube-api-access-bw558\") pod \"distinct-w0-f4de10-kserve-c47695d9f-fxcqb\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") " pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.290131 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.290049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:22.567445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:22.567419    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"]
Sep 11 01:19:22.569177 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:19:22.569151    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ae37207_ef77_44de_9811_b131dc58f8e0.slice/crio-d5592cd1055886aa54f6ab559f61ae271dfb9a16c676540477de5be54eecfae2 WatchSource:0}: Error finding container d5592cd1055886aa54f6ab559f61ae271dfb9a16c676540477de5be54eecfae2: Status 404 returned error can't find the container with id d5592cd1055886aa54f6ab559f61ae271dfb9a16c676540477de5be54eecfae2
Sep 11 01:19:23.502764 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:23.502731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb" event={"ID":"1ae37207-ef77-44de-9811-b131dc58f8e0","Type":"ContainerStarted","Data":"825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168"}
Sep 11 01:19:23.502764 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:23.502776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb" event={"ID":"1ae37207-ef77-44de-9811-b131dc58f8e0","Type":"ContainerStarted","Data":"d5592cd1055886aa54f6ab559f61ae271dfb9a16c676540477de5be54eecfae2"}
Sep 11 01:19:23.522441 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:23.522381    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb" podStartSLOduration=2.522360879 podStartE2EDuration="2.522360879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:19:23.521889881 +0000 UTC m=+1293.058541098" watchObservedRunningTime="2026-09-11 01:19:23.522360879 +0000 UTC m=+1293.059012079"
Sep 11 01:19:27.698252 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:27.698217    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:19:32.290716 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:32.290680    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:32.291165 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:32.290733    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:32.303186 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:32.303155    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:32.564486 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:32.564457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:19:51.496229 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:19:51.496193    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:20:11.281056 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:11.281024    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:20:24.778869 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.778777    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"]
Sep 11 01:20:24.782796 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.782776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.786064 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.786042    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4ec6e9\"/\"default-dockercfg-kg9nj\""
Sep 11 01:20:24.786202 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.786067    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4ec6e9\"/\"openshift-service-ca.crt\""
Sep 11 01:20:24.786202 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.786050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4ec6e9\"/\"noten-model-4ec6e9-kserve-self-signed-certs\""
Sep 11 01:20:24.786202 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.786045    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4ec6e9\"/\"kube-root-ca.crt\""
Sep 11 01:20:24.792901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.792882    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"]
Sep 11 01:20:24.855758 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.855730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2m7\" (UniqueName: \"kubernetes.io/projected/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-kube-api-access-qt2m7\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.855758 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.855762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-dshm\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.856031 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.855779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tmp-dir\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.856031 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.855903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-model-cache\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.856145 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.856042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tls-certs\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.856145 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.856126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-home\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957079 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tls-certs\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-home\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2m7\" (UniqueName: \"kubernetes.io/projected/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-kube-api-access-qt2m7\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-dshm\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tmp-dir\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-model-cache\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957725 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tmp-dir\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957858 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-home\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.957858 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.957755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-model-cache\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.959918 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.959893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-dshm\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.960252 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.960225    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tls-certs\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:24.974746 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:24.974721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2m7\" (UniqueName: \"kubernetes.io/projected/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-kube-api-access-qt2m7\") pod \"noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") " pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:25.093402 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:25.093369    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:25.369928 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:25.369902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"]
Sep 11 01:20:25.371711 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:20:25.371684    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbaa730e_ad9a_4c1d_af69_e99e6e4249c8.slice/crio-a992b7fad092609e1f8d7dc56e64322bf11d8ab62209207262376b51a4cea06e WatchSource:0}: Error finding container a992b7fad092609e1f8d7dc56e64322bf11d8ab62209207262376b51a4cea06e: Status 404 returned error can't find the container with id a992b7fad092609e1f8d7dc56e64322bf11d8ab62209207262376b51a4cea06e
Sep 11 01:20:25.842369 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:25.842331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8" event={"ID":"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8","Type":"ContainerStarted","Data":"92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004"}
Sep 11 01:20:25.842805 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:25.842377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8" event={"ID":"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8","Type":"ContainerStarted","Data":"a992b7fad092609e1f8d7dc56e64322bf11d8ab62209207262376b51a4cea06e"}
Sep 11 01:20:25.869833 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:25.869786    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8" podStartSLOduration=1.869771202 podStartE2EDuration="1.869771202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:20:25.868135499 +0000 UTC m=+1355.404786700" watchObservedRunningTime="2026-09-11 01:20:25.869771202 +0000 UTC m=+1355.406422403"
Sep 11 01:20:27.279629 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:27.279595    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"]
Sep 11 01:20:27.281547 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:27.281524    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e98d1e-29818155-87v7v"]
Sep 11 01:20:29.036886 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:29.036856    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99348cc8-24e0-4db1-a390-d0d97c57e195" path="/var/lib/kubelet/pods/99348cc8-24e0-4db1-a390-d0d97c57e195/volumes"
Sep 11 01:20:35.094191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:35.094151    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:35.094191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:35.094198    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:35.106792 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:35.106771    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:35.905203 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:35.905174    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:20:42.676032 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:20:42.675993    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:21:03.304133 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:03.304098    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"]
Sep 11 01:21:04.275207 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:04.275173    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:21:18.323666 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.323623    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8" podUID="fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" containerName="main" containerID="cri-o://92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004" gracePeriod=45
Sep 11 01:21:18.573269 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.573244    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:21:18.669275 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669192    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt2m7\" (UniqueName: \"kubernetes.io/projected/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-kube-api-access-qt2m7\") pod \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") "
Sep 11 01:21:18.669275 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669257    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-dshm\") pod \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") "
Sep 11 01:21:18.669521 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669283    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tls-certs\") pod \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") "
Sep 11 01:21:18.669521 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669326    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-home\") pod \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") "
Sep 11 01:21:18.669521 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669352    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-model-cache\") pod \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") "
Sep 11 01:21:18.669521 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669431    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tmp-dir\") pod \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\" (UID: \"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8\") "
Sep 11 01:21:18.669738 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669646    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-home" (OuterVolumeSpecName: "home") pod "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" (UID: "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:21:18.669738 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669685    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" (UID: "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:21:18.669850 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669751    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" (UID: "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:21:18.669850 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.669758    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-home\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:21:18.671435 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.671413    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-kube-api-access-qt2m7" (OuterVolumeSpecName: "kube-api-access-qt2m7") pod "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" (UID: "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8"). InnerVolumeSpecName "kube-api-access-qt2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:21:18.671747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.671723    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" (UID: "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:21:18.671747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.671730    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-dshm" (OuterVolumeSpecName: "dshm") pod "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" (UID: "fbaa730e-ad9a-4c1d-af69-e99e6e4249c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:21:18.770520 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.770487    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tmp-dir\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:21:18.770520 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.770514    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt2m7\" (UniqueName: \"kubernetes.io/projected/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-kube-api-access-qt2m7\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:21:18.770520 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.770523    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-dshm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:21:18.770744 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.770531    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-tls-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:21:18.770744 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:18.770539    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8-model-cache\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:21:19.123133 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.123098    2583 generic.go:356] "Generic (PLEG): container finished" podID="fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" containerID="92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004" exitCode=0
Sep 11 01:21:19.123335 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.123154    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"
Sep 11 01:21:19.123335 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.123183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8" event={"ID":"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8","Type":"ContainerDied","Data":"92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004"}
Sep 11 01:21:19.123335 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.123240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8" event={"ID":"fbaa730e-ad9a-4c1d-af69-e99e6e4249c8","Type":"ContainerDied","Data":"a992b7fad092609e1f8d7dc56e64322bf11d8ab62209207262376b51a4cea06e"}
Sep 11 01:21:19.123335 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.123265    2583 scope.go:117] "RemoveContainer" containerID="92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004"
Sep 11 01:21:19.132880 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.132856    2583 scope.go:117] "RemoveContainer" containerID="92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004"
Sep 11 01:21:19.133159 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:21:19.133140    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004\": container with ID starting with 92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004 not found: ID does not exist" containerID="92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004"
Sep 11 01:21:19.133230 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.133167    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004"} err="failed to get container status \"92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004\": rpc error: code = NotFound desc = could not find container \"92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004\": container with ID starting with 92d5e6723e64a61243b274b567fad4608ea0fff60f52b5b0732d2895bb315004 not found: ID does not exist"
Sep 11 01:21:19.142159 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.142131    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"]
Sep 11 01:21:19.146155 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:19.146136    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4ec6e9/noten-model-4ec6e9-kserve-6f7f6b76c-hv7h8"]
Sep 11 01:21:20.679178 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:20.679147    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"]
Sep 11 01:21:20.679625 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:20.679422    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2" podUID="b1870644-b475-4d91-8e24-de649b329680" containerName="payload-processing" containerID="cri-o://430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc" gracePeriod=30
Sep 11 01:21:20.948983 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:20.948960    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:21:21.039174 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.039131    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" path="/var/lib/kubelet/pods/fbaa730e-ad9a-4c1d-af69-e99e6e4249c8/volumes"
Sep 11 01:21:21.094493 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.094468    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1870644-b475-4d91-8e24-de649b329680-plugins-config-volume\") pod \"b1870644-b475-4d91-8e24-de649b329680\" (UID: \"b1870644-b475-4d91-8e24-de649b329680\") "
Sep 11 01:21:21.094649 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.094556    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkpk5\" (UniqueName: \"kubernetes.io/projected/b1870644-b475-4d91-8e24-de649b329680-kube-api-access-vkpk5\") pod \"b1870644-b475-4d91-8e24-de649b329680\" (UID: \"b1870644-b475-4d91-8e24-de649b329680\") "
Sep 11 01:21:21.094877 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.094848    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1870644-b475-4d91-8e24-de649b329680-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1870644-b475-4d91-8e24-de649b329680" (UID: "b1870644-b475-4d91-8e24-de649b329680"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:21:21.096634 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.096611    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1870644-b475-4d91-8e24-de649b329680-kube-api-access-vkpk5" (OuterVolumeSpecName: "kube-api-access-vkpk5") pod "b1870644-b475-4d91-8e24-de649b329680" (UID: "b1870644-b475-4d91-8e24-de649b329680"). InnerVolumeSpecName "kube-api-access-vkpk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:21:21.138849 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.138817    2583 generic.go:356] "Generic (PLEG): container finished" podID="b1870644-b475-4d91-8e24-de649b329680" containerID="430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc" exitCode=0
Sep 11 01:21:21.138958 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.138866    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"
Sep 11 01:21:21.138958 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.138898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2" event={"ID":"b1870644-b475-4d91-8e24-de649b329680","Type":"ContainerDied","Data":"430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc"}
Sep 11 01:21:21.138958 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.138948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2" event={"ID":"b1870644-b475-4d91-8e24-de649b329680","Type":"ContainerDied","Data":"2a379e4953569c23e87464b1698b10845d3bfb0d80e2ad49eec6e6e8d6f7a235"}
Sep 11 01:21:21.139063 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.138970    2583 scope.go:117] "RemoveContainer" containerID="430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc"
Sep 11 01:21:21.148164 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.148136    2583 scope.go:117] "RemoveContainer" containerID="430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc"
Sep 11 01:21:21.148484 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:21:21.148461    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc\": container with ID starting with 430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc not found: ID does not exist" containerID="430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc"
Sep 11 01:21:21.148559 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.148495    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc"} err="failed to get container status \"430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc\": rpc error: code = NotFound desc = could not find container \"430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc\": container with ID starting with 430707a488d96e0134fc3b700ad4360ca563d42c65cfa618113da130ef83f0cc not found: ID does not exist"
Sep 11 01:21:21.159180 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.159162    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"]
Sep 11 01:21:21.165547 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.165525    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10a215-d478dc4dc-twwq2"]
Sep 11 01:21:21.195742 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.195682    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1870644-b475-4d91-8e24-de649b329680-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:21:21.195827 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:21.195811    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkpk5\" (UniqueName: \"kubernetes.io/projected/b1870644-b475-4d91-8e24-de649b329680-kube-api-access-vkpk5\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:21:23.037486 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:23.037451    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1870644-b475-4d91-8e24-de649b329680" path="/var/lib/kubelet/pods/b1870644-b475-4d91-8e24-de649b329680/volumes"
Sep 11 01:21:30.388562 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:30.388528    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:21:51.075697 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:51.075629    2583 scope.go:117] "RemoveContainer" containerID="fc4af76ddb41a9c2c1162bc52f3965e89482fa59326cfd42148110e6393e1766"
Sep 11 01:21:59.985573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:21:59.985538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:22:02.636049 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.636011    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"]
Sep 11 01:22:02.636668 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.636642    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1870644-b475-4d91-8e24-de649b329680" containerName="payload-processing"
Sep 11 01:22:02.636716 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.636672    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1870644-b475-4d91-8e24-de649b329680" containerName="payload-processing"
Sep 11 01:22:02.636716 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.636707    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" containerName="main"
Sep 11 01:22:02.636780 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.636715    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" containerName="main"
Sep 11 01:22:02.636825 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.636812    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1870644-b475-4d91-8e24-de649b329680" containerName="payload-processing"
Sep 11 01:22:02.636860 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.636835    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbaa730e-ad9a-4c1d-af69-e99e6e4249c8" containerName="main"
Sep 11 01:22:02.641518 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.641500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:02.643898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.643879    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b9930f-dockercfg-wqwtc\""
Sep 11 01:22:02.644554 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.644532    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b9930f\""
Sep 11 01:22:02.648045 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.648023    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"]
Sep 11 01:22:02.701921 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.701892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14698a22-6a8c-471c-a67e-8045c43816d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7\" (UID: \"14698a22-6a8c-471c-a67e-8045c43816d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:02.702041 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.701934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c94p\" (UniqueName: \"kubernetes.io/projected/14698a22-6a8c-471c-a67e-8045c43816d5-kube-api-access-7c94p\") pod \"payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7\" (UID: \"14698a22-6a8c-471c-a67e-8045c43816d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:02.803232 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.803190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14698a22-6a8c-471c-a67e-8045c43816d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7\" (UID: \"14698a22-6a8c-471c-a67e-8045c43816d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:02.803430 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.803242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c94p\" (UniqueName: \"kubernetes.io/projected/14698a22-6a8c-471c-a67e-8045c43816d5-kube-api-access-7c94p\") pod \"payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7\" (UID: \"14698a22-6a8c-471c-a67e-8045c43816d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:02.803910 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.803889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14698a22-6a8c-471c-a67e-8045c43816d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7\" (UID: \"14698a22-6a8c-471c-a67e-8045c43816d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:02.811554 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.811534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c94p\" (UniqueName: \"kubernetes.io/projected/14698a22-6a8c-471c-a67e-8045c43816d5-kube-api-access-7c94p\") pod \"payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7\" (UID: \"14698a22-6a8c-471c-a67e-8045c43816d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:02.955282 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:02.955203    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:03.227082 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:03.227051    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"]
Sep 11 01:22:03.228070 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:22:03.228040    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14698a22_6a8c_471c_a67e_8045c43816d5.slice/crio-15bc36e86d9403b9eedb65c5817ca05c21f9c494aa92bcb09f04737565bb6241 WatchSource:0}: Error finding container 15bc36e86d9403b9eedb65c5817ca05c21f9c494aa92bcb09f04737565bb6241: Status 404 returned error can't find the container with id 15bc36e86d9403b9eedb65c5817ca05c21f9c494aa92bcb09f04737565bb6241
Sep 11 01:22:03.369972 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:03.369932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7" event={"ID":"14698a22-6a8c-471c-a67e-8045c43816d5","Type":"ContainerStarted","Data":"16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a"}
Sep 11 01:22:03.370143 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:03.369986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7" event={"ID":"14698a22-6a8c-471c-a67e-8045c43816d5","Type":"ContainerStarted","Data":"15bc36e86d9403b9eedb65c5817ca05c21f9c494aa92bcb09f04737565bb6241"}
Sep 11 01:22:03.370143 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:03.370023    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:03.388531 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:03.388485    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7" podStartSLOduration=1.388471554 podStartE2EDuration="1.388471554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:22:03.387247274 +0000 UTC m=+1452.923898474" watchObservedRunningTime="2026-09-11 01:22:03.388471554 +0000 UTC m=+1452.925122773"
Sep 11 01:22:14.378321 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:14.378286    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:22:30.182226 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.182188    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"]
Sep 11 01:22:30.182878 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.182790    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv" podUID="e4681e69-f131-4726-aaf6-afc46024335b" containerName="payload-processing" containerID="cri-o://36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710" gracePeriod=30
Sep 11 01:22:30.429591 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.429561    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:22:30.509133 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.509049    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4681e69-f131-4726-aaf6-afc46024335b" containerID="36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710" exitCode=0
Sep 11 01:22:30.509133 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.509092    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"
Sep 11 01:22:30.509133 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.509114    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv" event={"ID":"e4681e69-f131-4726-aaf6-afc46024335b","Type":"ContainerDied","Data":"36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710"}
Sep 11 01:22:30.509384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.509155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv" event={"ID":"e4681e69-f131-4726-aaf6-afc46024335b","Type":"ContainerDied","Data":"6c497b427e13d341ef82fc8687cace1a3551c2298a2c3e9ec8757c22fb32e80c"}
Sep 11 01:22:30.509384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.509181    2583 scope.go:117] "RemoveContainer" containerID="36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710"
Sep 11 01:22:30.521261 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.521238    2583 scope.go:117] "RemoveContainer" containerID="36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710"
Sep 11 01:22:30.521762 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:22:30.521736    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710\": container with ID starting with 36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710 not found: ID does not exist" containerID="36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710"
Sep 11 01:22:30.521898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.521776    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710"} err="failed to get container status \"36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710\": rpc error: code = NotFound desc = could not find container \"36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710\": container with ID starting with 36d2fa114b0a81e2a167d6b88b63efad78dbd31160c36ca33d9c711973a19710 not found: ID does not exist"
Sep 11 01:22:30.574748 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.574718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6frld\" (UniqueName: \"kubernetes.io/projected/e4681e69-f131-4726-aaf6-afc46024335b-kube-api-access-6frld\") pod \"e4681e69-f131-4726-aaf6-afc46024335b\" (UID: \"e4681e69-f131-4726-aaf6-afc46024335b\") "
Sep 11 01:22:30.574908 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.574762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4681e69-f131-4726-aaf6-afc46024335b-plugins-config-volume\") pod \"e4681e69-f131-4726-aaf6-afc46024335b\" (UID: \"e4681e69-f131-4726-aaf6-afc46024335b\") "
Sep 11 01:22:30.575157 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.575131    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4681e69-f131-4726-aaf6-afc46024335b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4681e69-f131-4726-aaf6-afc46024335b" (UID: "e4681e69-f131-4726-aaf6-afc46024335b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:22:30.576815 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.576788    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4681e69-f131-4726-aaf6-afc46024335b-kube-api-access-6frld" (OuterVolumeSpecName: "kube-api-access-6frld") pod "e4681e69-f131-4726-aaf6-afc46024335b" (UID: "e4681e69-f131-4726-aaf6-afc46024335b"). InnerVolumeSpecName "kube-api-access-6frld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:22:30.682752 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.676836    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6frld\" (UniqueName: \"kubernetes.io/projected/e4681e69-f131-4726-aaf6-afc46024335b-kube-api-access-6frld\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:22:30.682752 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.676890    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4681e69-f131-4726-aaf6-afc46024335b-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:22:30.829188 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.829164    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"]
Sep 11 01:22:30.835621 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:30.835596    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6f69f6e6-79bf759489-llzpv"]
Sep 11 01:22:31.037737 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:31.037709    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4681e69-f131-4726-aaf6-afc46024335b" path="/var/lib/kubelet/pods/e4681e69-f131-4726-aaf6-afc46024335b/volumes"
Sep 11 01:22:33.582606 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:33.582573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:22:50.969326 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:50.969287    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:22:50.971417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:50.971398    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:22:50.974761 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:50.974744    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:22:50.977012 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:22:50.976996    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:23:01.589094 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:01.589060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:23:27.212864 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:27.212777    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"]
Sep 11 01:23:27.213344 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:27.212991    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy" containerID="cri-o://bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace" gracePeriod=30
Sep 11 01:23:30.680586 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:30.680546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:23:32.476099 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.476043    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:23:32.557147 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557109    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgnlg\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-kube-api-access-tgnlg\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557336 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557156    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-data\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557336 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557185    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-envoy\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557336 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557209    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-certs\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557336 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-token\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557336 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557268    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-podinfo\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557336 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557292    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23685a78-f9fb-4132-bb71-7980c8d81f1f-istiod-ca-cert\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557372    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-credential-socket\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557436    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-socket\") pod \"23685a78-f9fb-4132-bb71-7980c8d81f1f\" (UID: \"23685a78-f9fb-4132-bb71-7980c8d81f1f\") "
Sep 11 01:23:32.557654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557629    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:32.557808 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557661    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:32.557808 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557717    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.557911 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.557849    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:32.558149 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.558107    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:32.558149 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.558150    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23685a78-f9fb-4132-bb71-7980c8d81f1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:32.559586 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.559559    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-kube-api-access-tgnlg" (OuterVolumeSpecName: "kube-api-access-tgnlg") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "kube-api-access-tgnlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:32.559935 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.559900    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:32.559935 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.559911    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:32.560093 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.560045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23685a78-f9fb-4132-bb71-7980c8d81f1f" (UID: "23685a78-f9fb-4132-bb71-7980c8d81f1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:23:32.659151 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.659112    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.659151 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.659145    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.659151 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.659155    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgnlg\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-kube-api-access-tgnlg\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.659399 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.659165    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.659399 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.659174    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.659399 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.659181    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.659399 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.659188    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23685a78-f9fb-4132-bb71-7980c8d81f1f-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.659399 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.659195    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23685a78-f9fb-4132-bb71-7980c8d81f1f-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:32.834414 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.834378    2583 generic.go:356] "Generic (PLEG): container finished" podID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerID="bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace" exitCode=0
Sep 11 01:23:32.834594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.834435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" event={"ID":"23685a78-f9fb-4132-bb71-7980c8d81f1f","Type":"ContainerDied","Data":"bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace"}
Sep 11 01:23:32.834594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.834440    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"
Sep 11 01:23:32.834594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.834508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b" event={"ID":"23685a78-f9fb-4132-bb71-7980c8d81f1f","Type":"ContainerDied","Data":"e3c856827139e3a420940bec06d39e1acf5e60a3878bde4404ffc68bd43944ea"}
Sep 11 01:23:32.834594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.834531    2583 scope.go:117] "RemoveContainer" containerID="bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace"
Sep 11 01:23:32.844631 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.844616    2583 scope.go:117] "RemoveContainer" containerID="6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05"
Sep 11 01:23:32.852427 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.852408    2583 scope.go:117] "RemoveContainer" containerID="bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace"
Sep 11 01:23:32.852696 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:23:32.852677    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace\": container with ID starting with bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace not found: ID does not exist" containerID="bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace"
Sep 11 01:23:32.852747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.852703    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace"} err="failed to get container status \"bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace\": rpc error: code = NotFound desc = could not find container \"bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace\": container with ID starting with bcf933512e05fb4549f6056559412dc313410209b56037eca13161088b162ace not found: ID does not exist"
Sep 11 01:23:32.852747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.852721    2583 scope.go:117] "RemoveContainer" containerID="6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05"
Sep 11 01:23:32.853003 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:23:32.852974    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05\": container with ID starting with 6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05 not found: ID does not exist" containerID="6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05"
Sep 11 01:23:32.853087 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.853011    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05"} err="failed to get container status \"6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05\": rpc error: code = NotFound desc = could not find container \"6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05\": container with ID starting with 6d0539e46020bbee31ebc7934eacf803ff33d4bda263bdc44b89e060f05b4a05 not found: ID does not exist"
Sep 11 01:23:32.855531 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.855511    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"]
Sep 11 01:23:32.859791 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:32.859770    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6f69f6e6-openshift-default-697758b5bd-xdb9b"]
Sep 11 01:23:33.036758 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:33.036723    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" path="/var/lib/kubelet/pods/23685a78-f9fb-4132-bb71-7980c8d81f1f/volumes"
Sep 11 01:23:43.658898 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:43.658858    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"]
Sep 11 01:23:43.680384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:43.680353    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"]
Sep 11 01:23:54.932965 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:54.932918    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"]
Sep 11 01:23:54.933437 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:54.933169    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k" podUID="19e3f2ae-3aa4-4b66-8142-e82e167b2148" containerName="maas-api" containerID="cri-o://39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310" gracePeriod=30
Sep 11 01:23:55.401544 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.401519    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:23:55.572094 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.572064    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e3f2ae-3aa4-4b66-8142-e82e167b2148-openshift-service-ca\") pod \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") "
Sep 11 01:23:55.572244 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.572137    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls\") pod \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") "
Sep 11 01:23:55.572244 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.572162    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb4dm\" (UniqueName: \"kubernetes.io/projected/19e3f2ae-3aa4-4b66-8142-e82e167b2148-kube-api-access-wb4dm\") pod \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") "
Sep 11 01:23:55.572244 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.572239    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-metrics-certs\") pod \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\" (UID: \"19e3f2ae-3aa4-4b66-8142-e82e167b2148\") "
Sep 11 01:23:55.572537 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.572511    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e3f2ae-3aa4-4b66-8142-e82e167b2148-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19e3f2ae-3aa4-4b66-8142-e82e167b2148" (UID: "19e3f2ae-3aa4-4b66-8142-e82e167b2148"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:23:55.574392 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.574359    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "19e3f2ae-3aa4-4b66-8142-e82e167b2148" (UID: "19e3f2ae-3aa4-4b66-8142-e82e167b2148"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:23:55.574489 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.574427    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e3f2ae-3aa4-4b66-8142-e82e167b2148-kube-api-access-wb4dm" (OuterVolumeSpecName: "kube-api-access-wb4dm") pod "19e3f2ae-3aa4-4b66-8142-e82e167b2148" (UID: "19e3f2ae-3aa4-4b66-8142-e82e167b2148"). InnerVolumeSpecName "kube-api-access-wb4dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:55.574540 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.574522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19e3f2ae-3aa4-4b66-8142-e82e167b2148" (UID: "19e3f2ae-3aa4-4b66-8142-e82e167b2148"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:23:55.673690 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.673651    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e3f2ae-3aa4-4b66-8142-e82e167b2148-openshift-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:55.673690 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.673684    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-maas-api-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:55.673690 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.673697    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4dm\" (UniqueName: \"kubernetes.io/projected/19e3f2ae-3aa4-4b66-8142-e82e167b2148-kube-api-access-wb4dm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:55.673923 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.673708    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e3f2ae-3aa4-4b66-8142-e82e167b2148-metrics-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:55.961596 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.961516    2583 generic.go:356] "Generic (PLEG): container finished" podID="19e3f2ae-3aa4-4b66-8142-e82e167b2148" containerID="39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310" exitCode=0
Sep 11 01:23:55.961596 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.961557    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"
Sep 11 01:23:55.962038 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.961596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k" event={"ID":"19e3f2ae-3aa4-4b66-8142-e82e167b2148","Type":"ContainerDied","Data":"39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310"}
Sep 11 01:23:55.962038 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.961647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k" event={"ID":"19e3f2ae-3aa4-4b66-8142-e82e167b2148","Type":"ContainerDied","Data":"6bce2f5aceab02c1975b98c6374b6599edb5cd1976228a58fba4b7cad05c584f"}
Sep 11 01:23:55.962038 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.961670    2583 scope.go:117] "RemoveContainer" containerID="39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310"
Sep 11 01:23:55.971532 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.971517    2583 scope.go:117] "RemoveContainer" containerID="39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310"
Sep 11 01:23:55.971812 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:23:55.971782    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310\": container with ID starting with 39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310 not found: ID does not exist" containerID="39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310"
Sep 11 01:23:55.971871 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.971820    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310"} err="failed to get container status \"39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310\": rpc error: code = NotFound desc = could not find container \"39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310\": container with ID starting with 39bda76392e7000e4189cf972eac2e69f09312b053808bed67245969642d2310 not found: ID does not exist"
Sep 11 01:23:55.994951 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:55.994924    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"]
Sep 11 01:23:56.004341 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:56.004322    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f4de10-6d86f67554-bb54k"]
Sep 11 01:23:57.037234 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:57.037200    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e3f2ae-3aa4-4b66-8142-e82e167b2148" path="/var/lib/kubelet/pods/19e3f2ae-3aa4-4b66-8142-e82e167b2148/volumes"
Sep 11 01:23:58.680644 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.680603    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb" podUID="1ae37207-ef77-44de-9811-b131dc58f8e0" containerName="main" containerID="cri-o://825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168" gracePeriod=45
Sep 11 01:23:58.703709 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.703672    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8" podUID="cc6db59b-0137-4c32-9a48-4a03c79a93d9" containerName="main" containerID="cri-o://569b04f3c44cdcbdd46359e326fe0c68f7e916b21cec49f1b681f6d7cec82167" gracePeriod=45
Sep 11 01:23:58.954228 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.954200    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:23:58.981215 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.980792    2583 generic.go:356] "Generic (PLEG): container finished" podID="cc6db59b-0137-4c32-9a48-4a03c79a93d9" containerID="569b04f3c44cdcbdd46359e326fe0c68f7e916b21cec49f1b681f6d7cec82167" exitCode=0
Sep 11 01:23:58.981215 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.980901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8" event={"ID":"cc6db59b-0137-4c32-9a48-4a03c79a93d9","Type":"ContainerDied","Data":"569b04f3c44cdcbdd46359e326fe0c68f7e916b21cec49f1b681f6d7cec82167"}
Sep 11 01:23:58.981215 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.980937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8" event={"ID":"cc6db59b-0137-4c32-9a48-4a03c79a93d9","Type":"ContainerDied","Data":"6be06eda2fbc88dad65ae23babc338160a938c8b8ac5c0ef451eea1a3abc2848"}
Sep 11 01:23:58.981215 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.980952    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be06eda2fbc88dad65ae23babc338160a938c8b8ac5c0ef451eea1a3abc2848"
Sep 11 01:23:58.982649 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.982629    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ae37207-ef77-44de-9811-b131dc58f8e0" containerID="825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168" exitCode=0
Sep 11 01:23:58.982744 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.982678    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"
Sep 11 01:23:58.982793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.982680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb" event={"ID":"1ae37207-ef77-44de-9811-b131dc58f8e0","Type":"ContainerDied","Data":"825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168"}
Sep 11 01:23:58.982793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.982782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb" event={"ID":"1ae37207-ef77-44de-9811-b131dc58f8e0","Type":"ContainerDied","Data":"d5592cd1055886aa54f6ab559f61ae271dfb9a16c676540477de5be54eecfae2"}
Sep 11 01:23:58.982871 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.982801    2583 scope.go:117] "RemoveContainer" containerID="825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168"
Sep 11 01:23:58.983673 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.983656    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:23:58.993849 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.993828    2583 scope.go:117] "RemoveContainer" containerID="825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168"
Sep 11 01:23:58.994165 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:23:58.994137    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168\": container with ID starting with 825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168 not found: ID does not exist" containerID="825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168"
Sep 11 01:23:58.994264 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:58.994174    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168"} err="failed to get container status \"825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168\": rpc error: code = NotFound desc = could not find container \"825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168\": container with ID starting with 825c60974f941c2c4c40c2aa5a077f031fd5d04068dddbc8e2494011125da168 not found: ID does not exist"
Sep 11 01:23:59.109753 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.109723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tls-certs\") pod \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") "
Sep 11 01:23:59.109753 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.109753    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae37207-ef77-44de-9811-b131dc58f8e0-tls-certs\") pod \"1ae37207-ef77-44de-9811-b131dc58f8e0\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") "
Sep 11 01:23:59.109960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.109775    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-dshm\") pod \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") "
Sep 11 01:23:59.109960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.109808    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-model-cache\") pod \"1ae37207-ef77-44de-9811-b131dc58f8e0\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") "
Sep 11 01:23:59.109960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.109839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h5d6\" (UniqueName: \"kubernetes.io/projected/cc6db59b-0137-4c32-9a48-4a03c79a93d9-kube-api-access-4h5d6\") pod \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") "
Sep 11 01:23:59.109960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.109870    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tmp-dir\") pod \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") "
Sep 11 01:23:59.109960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.109952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-dshm\") pod \"1ae37207-ef77-44de-9811-b131dc58f8e0\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") "
Sep 11 01:23:59.110227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.109994    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-model-cache\") pod \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") "
Sep 11 01:23:59.110227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110025    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw558\" (UniqueName: \"kubernetes.io/projected/1ae37207-ef77-44de-9811-b131dc58f8e0-kube-api-access-bw558\") pod \"1ae37207-ef77-44de-9811-b131dc58f8e0\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") "
Sep 11 01:23:59.110227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-tmp-dir\") pod \"1ae37207-ef77-44de-9811-b131dc58f8e0\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") "
Sep 11 01:23:59.110227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110084    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-home\") pod \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\" (UID: \"cc6db59b-0137-4c32-9a48-4a03c79a93d9\") "
Sep 11 01:23:59.110227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110104    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ae37207-ef77-44de-9811-b131dc58f8e0" (UID: "1ae37207-ef77-44de-9811-b131dc58f8e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:59.110227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110138    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-home\") pod \"1ae37207-ef77-44de-9811-b131dc58f8e0\" (UID: \"1ae37207-ef77-44de-9811-b131dc58f8e0\") "
Sep 11 01:23:59.110575 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110230    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc6db59b-0137-4c32-9a48-4a03c79a93d9" (UID: "cc6db59b-0137-4c32-9a48-4a03c79a93d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:59.110575 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ae37207-ef77-44de-9811-b131dc58f8e0" (UID: "1ae37207-ef77-44de-9811-b131dc58f8e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:59.110575 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110282    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc6db59b-0137-4c32-9a48-4a03c79a93d9" (UID: "cc6db59b-0137-4c32-9a48-4a03c79a93d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:59.110802 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110607    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-home" (OuterVolumeSpecName: "home") pod "cc6db59b-0137-4c32-9a48-4a03c79a93d9" (UID: "cc6db59b-0137-4c32-9a48-4a03c79a93d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:59.110938 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110866    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-home" (OuterVolumeSpecName: "home") pod "1ae37207-ef77-44de-9811-b131dc58f8e0" (UID: "1ae37207-ef77-44de-9811-b131dc58f8e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:59.111026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110973    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-tmp-dir\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.111026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.110993    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-home\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.111026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.111005    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-home\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.111026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.111017    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-model-cache\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.111189 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.111031    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tmp-dir\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.111189 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.111042    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-model-cache\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.112145 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.112115    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc6db59b-0137-4c32-9a48-4a03c79a93d9" (UID: "cc6db59b-0137-4c32-9a48-4a03c79a93d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:23:59.112494 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.112455    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6db59b-0137-4c32-9a48-4a03c79a93d9-kube-api-access-4h5d6" (OuterVolumeSpecName: "kube-api-access-4h5d6") pod "cc6db59b-0137-4c32-9a48-4a03c79a93d9" (UID: "cc6db59b-0137-4c32-9a48-4a03c79a93d9"). InnerVolumeSpecName "kube-api-access-4h5d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:59.112494 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.112479    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-dshm" (OuterVolumeSpecName: "dshm") pod "cc6db59b-0137-4c32-9a48-4a03c79a93d9" (UID: "cc6db59b-0137-4c32-9a48-4a03c79a93d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:59.112690 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.112664    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae37207-ef77-44de-9811-b131dc58f8e0-kube-api-access-bw558" (OuterVolumeSpecName: "kube-api-access-bw558") pod "1ae37207-ef77-44de-9811-b131dc58f8e0" (UID: "1ae37207-ef77-44de-9811-b131dc58f8e0"). InnerVolumeSpecName "kube-api-access-bw558". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:23:59.112751 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.112720    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae37207-ef77-44de-9811-b131dc58f8e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ae37207-ef77-44de-9811-b131dc58f8e0" (UID: "1ae37207-ef77-44de-9811-b131dc58f8e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:23:59.113076 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.113060    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-dshm" (OuterVolumeSpecName: "dshm") pod "1ae37207-ef77-44de-9811-b131dc58f8e0" (UID: "1ae37207-ef77-44de-9811-b131dc58f8e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:23:59.212171 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.212080    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6db59b-0137-4c32-9a48-4a03c79a93d9-tls-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.212171 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.212108    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae37207-ef77-44de-9811-b131dc58f8e0-tls-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.212171 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.212119    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6db59b-0137-4c32-9a48-4a03c79a93d9-dshm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.212171 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.212130    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5d6\" (UniqueName: \"kubernetes.io/projected/cc6db59b-0137-4c32-9a48-4a03c79a93d9-kube-api-access-4h5d6\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.212171 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.212143    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae37207-ef77-44de-9811-b131dc58f8e0-dshm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.212171 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.212154    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw558\" (UniqueName: \"kubernetes.io/projected/1ae37207-ef77-44de-9811-b131dc58f8e0-kube-api-access-bw558\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:23:59.305386 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.305355    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"]
Sep 11 01:23:59.309215 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.309189    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/distinct-w0-f4de10-kserve-c47695d9f-fxcqb"]
Sep 11 01:23:59.988528 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:23:59.988446    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"
Sep 11 01:24:00.007926 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:00.007900    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"]
Sep 11 01:24:00.011288 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:00.011263    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f4de10/simulator-w0-f4de10-kserve-5b9459b4b9-bckk8"]
Sep 11 01:24:01.039268 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:01.039230    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae37207-ef77-44de-9811-b131dc58f8e0" path="/var/lib/kubelet/pods/1ae37207-ef77-44de-9811-b131dc58f8e0/volumes"
Sep 11 01:24:01.039909 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:01.039889    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6db59b-0137-4c32-9a48-4a03c79a93d9" path="/var/lib/kubelet/pods/cc6db59b-0137-4c32-9a48-4a03c79a93d9/volumes"
Sep 11 01:24:01.279532 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:01.279499    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:24:22.001796 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.001764    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"]
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002185    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e3f2ae-3aa4-4b66-8142-e82e167b2148" containerName="maas-api"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002196    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e3f2ae-3aa4-4b66-8142-e82e167b2148" containerName="maas-api"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002213    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae37207-ef77-44de-9811-b131dc58f8e0" containerName="main"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002222    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae37207-ef77-44de-9811-b131dc58f8e0" containerName="main"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002230    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6db59b-0137-4c32-9a48-4a03c79a93d9" containerName="main"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002235    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6db59b-0137-4c32-9a48-4a03c79a93d9" containerName="main"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002240    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002245    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002251    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002257    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002273    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4681e69-f131-4726-aaf6-afc46024335b" containerName="payload-processing"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002279    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4681e69-f131-4726-aaf6-afc46024335b" containerName="payload-processing"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002363    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e3f2ae-3aa4-4b66-8142-e82e167b2148" containerName="maas-api"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002372    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4681e69-f131-4726-aaf6-afc46024335b" containerName="payload-processing"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002386    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002392    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae37207-ef77-44de-9811-b131dc58f8e0" containerName="main"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002398    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="23685a78-f9fb-4132-bb71-7980c8d81f1f" containerName="istio-proxy"
Sep 11 01:24:22.004048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.002405    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6db59b-0137-4c32-9a48-4a03c79a93d9" containerName="main"
Sep 11 01:24:22.005248 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.005233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.007341 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.007300    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f729f8-openshift-default-dockercfg-nsbkm\""
Sep 11 01:24:22.015010 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.014986    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"]
Sep 11 01:24:22.117817 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.117783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-envoy\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.117989 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.117833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-data\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.117989 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.117888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-credential-socket\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.117989 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.117936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtk8r\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-kube-api-access-wtk8r\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.118109 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.117984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istiod-ca-cert\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.118109 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.118046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-certs\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.118109 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.118070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-podinfo\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.118225 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.118121    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-socket\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.118225 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.118187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-token\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.218911 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.218880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-token\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219103 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.218931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-envoy\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219103 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.218950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-data\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219103 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.218969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-credential-socket\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219103 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.218988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtk8r\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-kube-api-access-wtk8r\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219103 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.219009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istiod-ca-cert\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219103 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.219028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-certs\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219103 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.219049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-podinfo\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219507 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.219225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-socket\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219649 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.219623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-data\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219851 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.219792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-certs\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219978 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.219886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-credential-socket\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.219978 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.219892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-socket\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.220094 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.220020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istiod-ca-cert\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.221859 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.221837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-envoy\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.221960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.221889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-podinfo\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.226915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.226895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-token\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.227014 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.226937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtk8r\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-kube-api-access-wtk8r\") pod \"e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") " pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.315198 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.315170    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:22.594514 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.594488    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"]
Sep 11 01:24:22.596444 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:24:22.596414    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8339de5b_ae7f_4bb6_b917_9f0e0919f92e.slice/crio-72e2516f789a0d38ff18d849584f4df23ad8520d3af03b8c151d59debcbf0c7d WatchSource:0}: Error finding container 72e2516f789a0d38ff18d849584f4df23ad8520d3af03b8c151d59debcbf0c7d: Status 404 returned error can't find the container with id 72e2516f789a0d38ff18d849584f4df23ad8520d3af03b8c151d59debcbf0c7d
Sep 11 01:24:22.598675 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.598644    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:22.598763 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.598724    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:22.598806 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:22.598767    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:23.103698 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:23.103665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" event={"ID":"8339de5b-ae7f-4bb6-b917-9f0e0919f92e","Type":"ContainerStarted","Data":"c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8"}
Sep 11 01:24:23.104093 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:23.103711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" event={"ID":"8339de5b-ae7f-4bb6-b917-9f0e0919f92e","Type":"ContainerStarted","Data":"72e2516f789a0d38ff18d849584f4df23ad8520d3af03b8c151d59debcbf0c7d"}
Sep 11 01:24:23.124615 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:23.124568    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" podStartSLOduration=2.124551843 podStartE2EDuration="2.124551843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:24:23.122688269 +0000 UTC m=+1592.659339471" watchObservedRunningTime="2026-09-11 01:24:23.124551843 +0000 UTC m=+1592.661203044"
Sep 11 01:24:23.315495 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:23.315465    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:23.320884 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:23.320858    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:24.108655 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:24.108624    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:24.110059 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:24.110033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:25.979511 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:25.979477    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:24:43.576416 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:43.576338    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:24:45.229543 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:45.229509    2583 generic.go:356] "Generic (PLEG): container finished" podID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerID="c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8" exitCode=0
Sep 11 01:24:45.230097 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:45.229583    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" event={"ID":"8339de5b-ae7f-4bb6-b917-9f0e0919f92e","Type":"ContainerDied","Data":"c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8"}
Sep 11 01:24:45.230097 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:45.229966    2583 scope.go:117] "RemoveContainer" containerID="c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8"
Sep 11 01:24:45.231103 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:45.231078    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:45.231180 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:45.231145    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:45.231180 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:45.231177    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:46.237005 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:46.236963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" event={"ID":"8339de5b-ae7f-4bb6-b917-9f0e0919f92e","Type":"ContainerStarted","Data":"ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc"}
Sep 11 01:24:46.315507 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:46.315471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:46.315778 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:46.315755    2583 patch_prober.go:28] interesting pod/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 01:24:46.315825 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:46.315802    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 01:24:47.315849 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:47.315821    2583 patch_prober.go:28] interesting pod/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 01:24:47.316207 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:47.315867    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 01:24:48.316102 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:48.316065    2583 patch_prober.go:28] interesting pod/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 01:24:48.316589 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:48.316150    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 01:24:49.316009 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:49.315975    2583 patch_prober.go:28] interesting pod/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 01:24:49.316189 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:49.316032    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 01:24:50.316293 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.316256    2583 patch_prober.go:28] interesting pod/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 01:24:50.316757 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.316342    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 01:24:50.818298 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.818267    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"]
Sep 11 01:24:50.829328 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.825080    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.855145 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.848324    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-be588a-openshift-default-dockercfg-p84s8\""
Sep 11 01:24:50.863883 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.863831    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"]
Sep 11 01:24:50.896803 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.896766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-envoy\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.896950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.896810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmgfs\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-kube-api-access-mmgfs\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.896950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.896878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-socket\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.896950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.896930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-podinfo\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.897126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.896959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-certs\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.897126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.896988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-data\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.897126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.897012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-token\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.897126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.897034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9592c11a-0202-47a8-8f48-e55ae5978e8f-istiod-ca-cert\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.897126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.897065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-credential-socket\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.998459 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-certs\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.998459 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-data\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.998719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-token\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.998719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9592c11a-0202-47a8-8f48-e55ae5978e8f-istiod-ca-cert\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.998719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-credential-socket\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.998879 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-envoy\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.998879 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmgfs\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-kube-api-access-mmgfs\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.998879 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-socket\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.999023 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-data\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.999023 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-certs\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.999023 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.998928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-podinfo\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.999173 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.999027    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-credential-socket\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.999249 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.999228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-socket\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:50.999568 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:50.999547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9592c11a-0202-47a8-8f48-e55ae5978e8f-istiod-ca-cert\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:51.001086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.001067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-envoy\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:51.001286 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.001267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-podinfo\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:51.005984 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.005955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-token\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:51.006079 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.006058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmgfs\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-kube-api-access-mmgfs\") pod \"e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") " pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:51.125572 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.125475    2583 scope.go:117] "RemoveContainer" containerID="569b04f3c44cdcbdd46359e326fe0c68f7e916b21cec49f1b681f6d7cec82167"
Sep 11 01:24:51.164152 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.164127    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:51.315597 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.315561    2583 patch_prober.go:28] interesting pod/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 01:24:51.315768 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.315619    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 01:24:51.450197 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.450164    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"]
Sep 11 01:24:51.452842 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:24:51.452815    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9592c11a_0202_47a8_8f48_e55ae5978e8f.slice/crio-104515573a771bbf535e614d254bf47e69cc2a2603fd965eda9ef00e6d811487 WatchSource:0}: Error finding container 104515573a771bbf535e614d254bf47e69cc2a2603fd965eda9ef00e6d811487: Status 404 returned error can't find the container with id 104515573a771bbf535e614d254bf47e69cc2a2603fd965eda9ef00e6d811487
Sep 11 01:24:51.455359 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.455328    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:51.455435 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.455401    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:51.455489 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:51.455433    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:24:52.273594 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:52.273557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" event={"ID":"9592c11a-0202-47a8-8f48-e55ae5978e8f","Type":"ContainerStarted","Data":"92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb"}
Sep 11 01:24:52.273766 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:52.273617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" event={"ID":"9592c11a-0202-47a8-8f48-e55ae5978e8f","Type":"ContainerStarted","Data":"104515573a771bbf535e614d254bf47e69cc2a2603fd965eda9ef00e6d811487"}
Sep 11 01:24:52.293879 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:52.293832    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" podStartSLOduration=2.293817054 podStartE2EDuration="2.293817054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:24:52.291580588 +0000 UTC m=+1621.828231792" watchObservedRunningTime="2026-09-11 01:24:52.293817054 +0000 UTC m=+1621.830468255"
Sep 11 01:24:52.320441 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:52.320411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:52.320812 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:52.320787    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:52.321789 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:52.321769    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:24:53.164386 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:53.164350    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:53.169702 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:53.169679    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:53.278482 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:53.278448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:53.279437 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:53.279416    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:24:55.389475 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.389443    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"]
Sep 11 01:24:55.392222 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.392191    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.394494 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.394470    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-be588a-serving-cert\""
Sep 11 01:24:55.404589 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.404561    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"]
Sep 11 01:24:55.441344 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.441266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.441576 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.441382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-metrics-certs\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.441576 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.441444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-openshift-service-ca\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.441688 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.441573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lfp7\" (UniqueName: \"kubernetes.io/projected/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-kube-api-access-4lfp7\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.542583 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.542549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lfp7\" (UniqueName: \"kubernetes.io/projected/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-kube-api-access-4lfp7\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.542768 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.542608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.542768 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.542629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-metrics-certs\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.542768 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.542657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-openshift-service-ca\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.542768 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:24:55.542770    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-serving-cert: secret "maas-api-e2e-api-b-be588a-serving-cert" not found
Sep 11 01:24:55.543012 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:24:55.542862    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls podName:4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57 nodeName:}" failed. No retries permitted until 2026-09-11 01:24:56.042838991 +0000 UTC m=+1625.579490189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls") pod "maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw" (UID: "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57") : secret "maas-api-e2e-api-b-be588a-serving-cert" not found
Sep 11 01:24:55.543404 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.543386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-openshift-service-ca\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.544970 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.544951    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-metrics-certs\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:55.554347 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:55.554299    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lfp7\" (UniqueName: \"kubernetes.io/projected/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-kube-api-access-4lfp7\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:56.048009 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:56.047961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:56.050461 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:56.050433    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls\") pod \"maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:56.304830 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:56.304753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:56.578526 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:56.578499    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"]
Sep 11 01:24:56.580264 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:24:56.580236    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4ce473_ffb2_4a6d_b64a_bbf33e9e4f57.slice/crio-6ca2b6125f82c1cc5e43b3bf2487948447d2af66a7276f17ea0e1554c633b34e WatchSource:0}: Error finding container 6ca2b6125f82c1cc5e43b3bf2487948447d2af66a7276f17ea0e1554c633b34e: Status 404 returned error can't find the container with id 6ca2b6125f82c1cc5e43b3bf2487948447d2af66a7276f17ea0e1554c633b34e
Sep 11 01:24:56.581837 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:56.581817    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 01:24:57.300785 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:57.300691    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw" event={"ID":"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57","Type":"ContainerStarted","Data":"a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565"}
Sep 11 01:24:57.300785 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:57.300748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:24:57.300785 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:57.300767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw" event={"ID":"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57","Type":"ContainerStarted","Data":"6ca2b6125f82c1cc5e43b3bf2487948447d2af66a7276f17ea0e1554c633b34e"}
Sep 11 01:24:57.321625 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:24:57.321576    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw" podStartSLOduration=1.969258785 podStartE2EDuration="2.321560154s" totalImagesPullingTime="352.301369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:24:57.318854333 +0000 UTC m=+1626.855505534" watchObservedRunningTime="2026-09-11 01:24:57.321560154 +0000 UTC m=+1626.858211354"
Sep 11 01:25:03.310582 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:03.310554    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:25:06.285556 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:06.285519    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:25:16.406651 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:16.406569    2583 generic.go:356] "Generic (PLEG): container finished" podID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerID="92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb" exitCode=0
Sep 11 01:25:16.406651 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:16.406605    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" event={"ID":"9592c11a-0202-47a8-8f48-e55ae5978e8f","Type":"ContainerDied","Data":"92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb"}
Sep 11 01:25:16.407139 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:16.406980    2583 scope.go:117] "RemoveContainer" containerID="92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb"
Sep 11 01:25:16.408080 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:16.408052    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:25:16.408173 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:16.408118    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:25:16.408173 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:16.408151    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:25:17.413353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:17.413288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" event={"ID":"9592c11a-0202-47a8-8f48-e55ae5978e8f","Type":"ContainerStarted","Data":"fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23"}
Sep 11 01:25:18.164377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:18.164345    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:25:18.164578 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:18.164541    2583 patch_prober.go:28] interesting pod/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 01:25:18.164666 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:18.164606    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 01:25:19.165290 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:19.165254    2583 patch_prober.go:28] interesting pod/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 01:25:19.165700 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:19.165339    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 01:25:20.165286 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:20.165248    2583 patch_prober.go:28] interesting pod/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 01:25:20.165482 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:20.165339    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 01:25:21.164872 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:21.164837    2583 patch_prober.go:28] interesting pod/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 01:25:21.165058 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:21.164890    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 01:25:22.165303 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:22.165266    2583 patch_prober.go:28] interesting pod/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 11 01:25:22.165789 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:22.165372    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 11 01:25:23.168968 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:23.168939    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:25:23.169374 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:23.169215    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:25:23.169821 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:23.169802    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:25:27.983425 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:27.983388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:25:45.025492 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:45.025449    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"]
Sep 11 01:25:45.026069 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:45.025696    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy" containerID="cri-o://ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc" gracePeriod=30
Sep 11 01:25:46.683600 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:46.683562    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:25:50.279700 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.279676    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:25:50.368196 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368168    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-credential-socket\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368392 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-token\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368392 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-certs\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368392 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368293    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtk8r\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-kube-api-access-wtk8r\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368392 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368339    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-podinfo\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368392 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368384    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-envoy\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368411    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istiod-ca-cert\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368446    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:25:50.368715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368454    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-data\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368578    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-socket\") pod \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\" (UID: \"8339de5b-ae7f-4bb6-b917-9f0e0919f92e\") "
Sep 11 01:25:50.368715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368584    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:25:50.368715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368687    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-data" (OuterVolumeSpecName: "istio-data") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:25:50.369026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:25:50.369026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368962    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:25:50.369026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368976    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.369026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.368993    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.369026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.369005    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.369026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.369018    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.370447 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.370420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-token" (OuterVolumeSpecName: "istio-token") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:25:50.370553 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.370502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:25:50.370596 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.370546    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:25:50.370596 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.370553    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-kube-api-access-wtk8r" (OuterVolumeSpecName: "kube-api-access-wtk8r") pod "8339de5b-ae7f-4bb6-b917-9f0e0919f92e" (UID: "8339de5b-ae7f-4bb6-b917-9f0e0919f92e"). InnerVolumeSpecName "kube-api-access-wtk8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:25:50.469769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.469745    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.469769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.469767    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtk8r\" (UniqueName: \"kubernetes.io/projected/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-kube-api-access-wtk8r\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.469920 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.469777    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.469920 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.469785    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.469920 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.469793    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8339de5b-ae7f-4bb6-b917-9f0e0919f92e-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:25:50.598791 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.598756    2583 generic.go:356] "Generic (PLEG): container finished" podID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerID="ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc" exitCode=0
Sep 11 01:25:50.598966 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.598810    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"
Sep 11 01:25:50.598966 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.598823    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" event={"ID":"8339de5b-ae7f-4bb6-b917-9f0e0919f92e","Type":"ContainerDied","Data":"ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc"}
Sep 11 01:25:50.598966 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.598866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn" event={"ID":"8339de5b-ae7f-4bb6-b917-9f0e0919f92e","Type":"ContainerDied","Data":"72e2516f789a0d38ff18d849584f4df23ad8520d3af03b8c151d59debcbf0c7d"}
Sep 11 01:25:50.598966 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.598889    2583 scope.go:117] "RemoveContainer" containerID="ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc"
Sep 11 01:25:50.609282 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.609257    2583 scope.go:117] "RemoveContainer" containerID="c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8"
Sep 11 01:25:50.617996 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.617978    2583 scope.go:117] "RemoveContainer" containerID="ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc"
Sep 11 01:25:50.618279 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:25:50.618240    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc\": container with ID starting with ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc not found: ID does not exist" containerID="ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc"
Sep 11 01:25:50.618279 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.618268    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc"} err="failed to get container status \"ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc\": rpc error: code = NotFound desc = could not find container \"ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc\": container with ID starting with ad5297ceeba81c0a020c37bde36efad4528b1cfd30840c90b72a24fdfc39e2dc not found: ID does not exist"
Sep 11 01:25:50.618380 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.618285    2583 scope.go:117] "RemoveContainer" containerID="c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8"
Sep 11 01:25:50.618662 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:25:50.618633    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8\": container with ID starting with c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8 not found: ID does not exist" containerID="c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8"
Sep 11 01:25:50.618707 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.618669    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8"} err="failed to get container status \"c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8\": rpc error: code = NotFound desc = could not find container \"c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8\": container with ID starting with c49560c3f3374c56a36629089df1f553feb621c2ab8e0a623185a2949bd627d8 not found: ID does not exist"
Sep 11 01:25:50.622467 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.622446    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"]
Sep 11 01:25:50.625529 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:50.625508    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f729f8-openshift-default-86f7f474dd-vghhn"]
Sep 11 01:25:51.044047 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:25:51.043980    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" path="/var/lib/kubelet/pods/8339de5b-ae7f-4bb6-b917-9f0e0919f92e/volumes"
Sep 11 01:26:00.956439 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:00.956400    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"]
Sep 11 01:26:00.957016 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:00.956633    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw" podUID="4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" containerName="maas-api" containerID="cri-o://a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565" gracePeriod=30
Sep 11 01:26:01.206257 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.206234    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:26:01.270598 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.270521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-metrics-certs\") pod \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") "
Sep 11 01:26:01.270598 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.270573    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls\") pod \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") "
Sep 11 01:26:01.270598 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.270595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-openshift-service-ca\") pod \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") "
Sep 11 01:26:01.270816 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.270777    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lfp7\" (UniqueName: \"kubernetes.io/projected/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-kube-api-access-4lfp7\") pod \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\" (UID: \"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57\") "
Sep 11 01:26:01.270954 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.270929    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" (UID: "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:26:01.271142 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.271127    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-openshift-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:01.272776 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.272744    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" (UID: "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:26:01.272776 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.272750    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-kube-api-access-4lfp7" (OuterVolumeSpecName: "kube-api-access-4lfp7") pod "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" (UID: "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57"). InnerVolumeSpecName "kube-api-access-4lfp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:26:01.272930 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.272832    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" (UID: "4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:26:01.372057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.372026    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-maas-api-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:01.372057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.372051    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lfp7\" (UniqueName: \"kubernetes.io/projected/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-kube-api-access-4lfp7\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:01.372057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.372062    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57-metrics-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:01.656412 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.656372    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" containerID="a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565" exitCode=0
Sep 11 01:26:01.656585 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.656440    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"
Sep 11 01:26:01.656585 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.656458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw" event={"ID":"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57","Type":"ContainerDied","Data":"a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565"}
Sep 11 01:26:01.656585 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.656537    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw" event={"ID":"4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57","Type":"ContainerDied","Data":"6ca2b6125f82c1cc5e43b3bf2487948447d2af66a7276f17ea0e1554c633b34e"}
Sep 11 01:26:01.656585 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.656561    2583 scope.go:117] "RemoveContainer" containerID="a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565"
Sep 11 01:26:01.665871 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.665854    2583 scope.go:117] "RemoveContainer" containerID="a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565"
Sep 11 01:26:01.666148 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:26:01.666127    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565\": container with ID starting with a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565 not found: ID does not exist" containerID="a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565"
Sep 11 01:26:01.666197 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.666157    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565"} err="failed to get container status \"a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565\": rpc error: code = NotFound desc = could not find container \"a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565\": container with ID starting with a781f903a98c1c761e680e8b466b176ebb337aa68b3749c1d6ce5d1c0c748565 not found: ID does not exist"
Sep 11 01:26:01.677034 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.677012    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"]
Sep 11 01:26:01.680935 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:01.680914    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-be588a-84779cbbd6-p8cxw"]
Sep 11 01:26:02.074988 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:02.074951    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:26:03.037815 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:03.037782    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" path="/var/lib/kubelet/pods/4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57/volumes"
Sep 11 01:26:07.628518 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:07.628485    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"]
Sep 11 01:26:07.629329 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:07.629278    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy" containerID="cri-o://fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23" gracePeriod=30
Sep 11 01:26:08.171576 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:08.171538    2583 patch_prober.go:28] interesting pod/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 01:26:08.171762 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:08.171602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 01:26:12.884590 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.884564    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:26:12.974620 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9592c11a-0202-47a8-8f48-e55ae5978e8f-istiod-ca-cert\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.974620 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974600    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-podinfo\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.974803 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974632    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-token\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.974803 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974711    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmgfs\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-kube-api-access-mmgfs\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.974803 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974779    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-data\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.974960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974832    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-envoy\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.974960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-socket\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.974960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974897    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-certs\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.974960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974946    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-credential-socket\") pod \"9592c11a-0202-47a8-8f48-e55ae5978e8f\" (UID: \"9592c11a-0202-47a8-8f48-e55ae5978e8f\") "
Sep 11 01:26:12.975172 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.975018    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-data" (OuterVolumeSpecName: "istio-data") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:12.975172 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.974964    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9592c11a-0202-47a8-8f48-e55ae5978e8f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:26:12.975282 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.975214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:12.975282 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.975243    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:12.975411 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.975320    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:12.975411 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.975342    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:12.975411 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.975356    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9592c11a-0202-47a8-8f48-e55ae5978e8f-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:12.976025 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.975344    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:12.977442 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.977406    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-kube-api-access-mmgfs" (OuterVolumeSpecName: "kube-api-access-mmgfs") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "kube-api-access-mmgfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:26:12.977442 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.977425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:26:12.983929 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.983907    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-token" (OuterVolumeSpecName: "istio-token") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:26:12.984170 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:12.984155    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9592c11a-0202-47a8-8f48-e55ae5978e8f" (UID: "9592c11a-0202-47a8-8f48-e55ae5978e8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:13.076741 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.076700    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:13.076741 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.076733    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:13.076741 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.076743    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9592c11a-0202-47a8-8f48-e55ae5978e8f-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:13.076967 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.076752    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:13.076967 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.076766    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:13.076967 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.076778    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmgfs\" (UniqueName: \"kubernetes.io/projected/9592c11a-0202-47a8-8f48-e55ae5978e8f-kube-api-access-mmgfs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:13.717353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.717292    2583 generic.go:356] "Generic (PLEG): container finished" podID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerID="fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23" exitCode=0
Sep 11 01:26:13.717526 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.717382    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"
Sep 11 01:26:13.717526 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.717376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" event={"ID":"9592c11a-0202-47a8-8f48-e55ae5978e8f","Type":"ContainerDied","Data":"fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23"}
Sep 11 01:26:13.717526 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.717497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz" event={"ID":"9592c11a-0202-47a8-8f48-e55ae5978e8f","Type":"ContainerDied","Data":"104515573a771bbf535e614d254bf47e69cc2a2603fd965eda9ef00e6d811487"}
Sep 11 01:26:13.717526 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.717520    2583 scope.go:117] "RemoveContainer" containerID="fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23"
Sep 11 01:26:13.727261 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.727241    2583 scope.go:117] "RemoveContainer" containerID="92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb"
Sep 11 01:26:13.735609 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.735591    2583 scope.go:117] "RemoveContainer" containerID="fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23"
Sep 11 01:26:13.735721 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.735705    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"]
Sep 11 01:26:13.735879 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:26:13.735860    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23\": container with ID starting with fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23 not found: ID does not exist" containerID="fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23"
Sep 11 01:26:13.735922 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.735886    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23"} err="failed to get container status \"fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23\": rpc error: code = NotFound desc = could not find container \"fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23\": container with ID starting with fda836c7106fb757e2a643123f29f3e0820860a9de5c05ad07395e6cb7451d23 not found: ID does not exist"
Sep 11 01:26:13.735922 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.735905    2583 scope.go:117] "RemoveContainer" containerID="92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb"
Sep 11 01:26:13.736174 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:26:13.736157    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb\": container with ID starting with 92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb not found: ID does not exist" containerID="92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb"
Sep 11 01:26:13.736226 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.736180    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb"} err="failed to get container status \"92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb\": rpc error: code = NotFound desc = could not find container \"92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb\": container with ID starting with 92380225d59556cc13eb824e92fa14d2fb8828ea0e9fec7e2cf882004bce0eeb not found: ID does not exist"
Sep 11 01:26:13.739535 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:13.739512    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-be588a-openshift-default-8f67f9ccb-ntqhz"]
Sep 11 01:26:15.037480 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:15.037449    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" path="/var/lib/kubelet/pods/9592c11a-0202-47a8-8f48-e55ae5978e8f/volumes"
Sep 11 01:26:18.673289 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:18.673254    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"]
Sep 11 01:26:18.673752 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:18.673525    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54" podUID="37cdab54-d747-4276-aced-bee307464f11" containerName="payload-processing" containerID="cri-o://bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579" gracePeriod=30
Sep 11 01:26:18.679200 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:18.679176    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"]
Sep 11 01:26:18.679418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:18.679379    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk" podUID="f09932b8-f03a-4618-a119-8b69c5b48569" containerName="payload-pre-processing" containerID="cri-o://8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501" gracePeriod=30
Sep 11 01:26:18.959605 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:18.959582    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:26:18.962158 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:18.962141    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:26:19.032584 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.032550    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2mzq\" (UniqueName: \"kubernetes.io/projected/37cdab54-d747-4276-aced-bee307464f11-kube-api-access-q2mzq\") pod \"37cdab54-d747-4276-aced-bee307464f11\" (UID: \"37cdab54-d747-4276-aced-bee307464f11\") "
Sep 11 01:26:19.032747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.032651    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cdab54-d747-4276-aced-bee307464f11-plugins-config-volume\") pod \"37cdab54-d747-4276-aced-bee307464f11\" (UID: \"37cdab54-d747-4276-aced-bee307464f11\") "
Sep 11 01:26:19.032747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.032672    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnxp8\" (UniqueName: \"kubernetes.io/projected/f09932b8-f03a-4618-a119-8b69c5b48569-kube-api-access-wnxp8\") pod \"f09932b8-f03a-4618-a119-8b69c5b48569\" (UID: \"f09932b8-f03a-4618-a119-8b69c5b48569\") "
Sep 11 01:26:19.032747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.032736    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09932b8-f03a-4618-a119-8b69c5b48569-plugins-config-volume\") pod \"f09932b8-f03a-4618-a119-8b69c5b48569\" (UID: \"f09932b8-f03a-4618-a119-8b69c5b48569\") "
Sep 11 01:26:19.033155 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.033123    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cdab54-d747-4276-aced-bee307464f11-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37cdab54-d747-4276-aced-bee307464f11" (UID: "37cdab54-d747-4276-aced-bee307464f11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:26:19.033155 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.033132    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09932b8-f03a-4618-a119-8b69c5b48569-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f09932b8-f03a-4618-a119-8b69c5b48569" (UID: "f09932b8-f03a-4618-a119-8b69c5b48569"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:26:19.034775 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.034747    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09932b8-f03a-4618-a119-8b69c5b48569-kube-api-access-wnxp8" (OuterVolumeSpecName: "kube-api-access-wnxp8") pod "f09932b8-f03a-4618-a119-8b69c5b48569" (UID: "f09932b8-f03a-4618-a119-8b69c5b48569"). InnerVolumeSpecName "kube-api-access-wnxp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:26:19.034899 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.034858    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cdab54-d747-4276-aced-bee307464f11-kube-api-access-q2mzq" (OuterVolumeSpecName: "kube-api-access-q2mzq") pod "37cdab54-d747-4276-aced-bee307464f11" (UID: "37cdab54-d747-4276-aced-bee307464f11"). InnerVolumeSpecName "kube-api-access-q2mzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:26:19.134283 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.134250    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cdab54-d747-4276-aced-bee307464f11-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:19.134283 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.134276    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnxp8\" (UniqueName: \"kubernetes.io/projected/f09932b8-f03a-4618-a119-8b69c5b48569-kube-api-access-wnxp8\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:19.134283 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.134286    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09932b8-f03a-4618-a119-8b69c5b48569-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:19.134511 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.134294    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2mzq\" (UniqueName: \"kubernetes.io/projected/37cdab54-d747-4276-aced-bee307464f11-kube-api-access-q2mzq\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:19.479864 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.479825    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:26:19.751552 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.751472    2583 generic.go:356] "Generic (PLEG): container finished" podID="f09932b8-f03a-4618-a119-8b69c5b48569" containerID="8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501" exitCode=0
Sep 11 01:26:19.751552 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.751528    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"
Sep 11 01:26:19.751972 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.751556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk" event={"ID":"f09932b8-f03a-4618-a119-8b69c5b48569","Type":"ContainerDied","Data":"8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501"}
Sep 11 01:26:19.751972 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.751611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk" event={"ID":"f09932b8-f03a-4618-a119-8b69c5b48569","Type":"ContainerDied","Data":"adc03f9a92a203aa2fc1a645021e8f1d8000360b1a930556e1e72e91643745ee"}
Sep 11 01:26:19.751972 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.751644    2583 scope.go:117] "RemoveContainer" containerID="8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501"
Sep 11 01:26:19.752968 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.752945    2583 generic.go:356] "Generic (PLEG): container finished" podID="37cdab54-d747-4276-aced-bee307464f11" containerID="bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579" exitCode=0
Sep 11 01:26:19.753020 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.752976    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54" event={"ID":"37cdab54-d747-4276-aced-bee307464f11","Type":"ContainerDied","Data":"bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579"}
Sep 11 01:26:19.753020 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.753009    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"
Sep 11 01:26:19.753091 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.753019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54" event={"ID":"37cdab54-d747-4276-aced-bee307464f11","Type":"ContainerDied","Data":"181c89c0ce798637aded554ac67c83478a2071e29249bf7968580fad0220e395"}
Sep 11 01:26:19.760674 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.760657    2583 scope.go:117] "RemoveContainer" containerID="8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501"
Sep 11 01:26:19.760957 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:26:19.760937    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501\": container with ID starting with 8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501 not found: ID does not exist" containerID="8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501"
Sep 11 01:26:19.761004 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.760964    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501"} err="failed to get container status \"8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501\": rpc error: code = NotFound desc = could not find container \"8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501\": container with ID starting with 8418549670d718894fa573a6af64cad84dfb593a4af4de9e87b240477d8e8501 not found: ID does not exist"
Sep 11 01:26:19.761004 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.760981    2583 scope.go:117] "RemoveContainer" containerID="bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579"
Sep 11 01:26:19.768535 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.768519    2583 scope.go:117] "RemoveContainer" containerID="bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579"
Sep 11 01:26:19.768824 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:26:19.768798    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579\": container with ID starting with bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579 not found: ID does not exist" containerID="bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579"
Sep 11 01:26:19.768924 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.768828    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579"} err="failed to get container status \"bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579\": rpc error: code = NotFound desc = could not find container \"bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579\": container with ID starting with bf7a927c32e41b77f943e9a7dc8b9327dbefba989b5bc58bf1a4f4a5dd89f579 not found: ID does not exist"
Sep 11 01:26:19.770294 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.770275    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"]
Sep 11 01:26:19.776369 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.776351    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-76bb56-5cdb6ffcc8-ntrpk"]
Sep 11 01:26:19.786195 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.786176    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"]
Sep 11 01:26:19.791949 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:19.791929    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-76bb56-85f969dc7-r9w54"]
Sep 11 01:26:21.036529 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:21.036498    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37cdab54-d747-4276-aced-bee307464f11" path="/var/lib/kubelet/pods/37cdab54-d747-4276-aced-bee307464f11/volumes"
Sep 11 01:26:21.036900 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:21.036849    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f09932b8-f03a-4618-a119-8b69c5b48569" path="/var/lib/kubelet/pods/f09932b8-f03a-4618-a119-8b69c5b48569/volumes"
Sep 11 01:26:29.577072 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:29.577036    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:26:36.588162 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.588126    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"]
Sep 11 01:26:36.588707 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.588375    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r" podUID="1642c6ef-7c39-41f8-86d4-d1f7780aca05" containerName="maas-api" containerID="cri-o://93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e" gracePeriod=30
Sep 11 01:26:36.841186 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.841109    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:26:36.846281 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.846251    2583 generic.go:356] "Generic (PLEG): container finished" podID="1642c6ef-7c39-41f8-86d4-d1f7780aca05" containerID="93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e" exitCode=0
Sep 11 01:26:36.846454 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.846290    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"
Sep 11 01:26:36.846454 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.846333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r" event={"ID":"1642c6ef-7c39-41f8-86d4-d1f7780aca05","Type":"ContainerDied","Data":"93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e"}
Sep 11 01:26:36.846454 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.846410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r" event={"ID":"1642c6ef-7c39-41f8-86d4-d1f7780aca05","Type":"ContainerDied","Data":"3cc6c17f7189d28400b8a28e973e3354869083607283e2a550c1f59ffb901ef9"}
Sep 11 01:26:36.846454 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.846434    2583 scope.go:117] "RemoveContainer" containerID="93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e"
Sep 11 01:26:36.854270 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.854254    2583 scope.go:117] "RemoveContainer" containerID="93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e"
Sep 11 01:26:36.854536 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:26:36.854519    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e\": container with ID starting with 93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e not found: ID does not exist" containerID="93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e"
Sep 11 01:26:36.854591 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.854543    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e"} err="failed to get container status \"93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e\": rpc error: code = NotFound desc = could not find container \"93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e\": container with ID starting with 93ea19396a36a5dacb71cb8080f77c03c2483b554104156aec4b6c5c05cf702e not found: ID does not exist"
Sep 11 01:26:36.992685 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.992651    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-maas-api-tls\") pod \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") "
Sep 11 01:26:36.992851 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.992699    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6jkx\" (UniqueName: \"kubernetes.io/projected/1642c6ef-7c39-41f8-86d4-d1f7780aca05-kube-api-access-p6jkx\") pod \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") "
Sep 11 01:26:36.992851 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.992729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-metrics-certs\") pod \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") "
Sep 11 01:26:36.992851 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.992775    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1642c6ef-7c39-41f8-86d4-d1f7780aca05-openshift-service-ca\") pod \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\" (UID: \"1642c6ef-7c39-41f8-86d4-d1f7780aca05\") "
Sep 11 01:26:36.993222 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.993194    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1642c6ef-7c39-41f8-86d4-d1f7780aca05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1642c6ef-7c39-41f8-86d4-d1f7780aca05" (UID: "1642c6ef-7c39-41f8-86d4-d1f7780aca05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:26:36.994861 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.994830    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1642c6ef-7c39-41f8-86d4-d1f7780aca05" (UID: "1642c6ef-7c39-41f8-86d4-d1f7780aca05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:26:36.994955 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.994867    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1642c6ef-7c39-41f8-86d4-d1f7780aca05" (UID: "1642c6ef-7c39-41f8-86d4-d1f7780aca05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:26:36.994955 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:36.994895    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1642c6ef-7c39-41f8-86d4-d1f7780aca05-kube-api-access-p6jkx" (OuterVolumeSpecName: "kube-api-access-p6jkx") pod "1642c6ef-7c39-41f8-86d4-d1f7780aca05" (UID: "1642c6ef-7c39-41f8-86d4-d1f7780aca05"). InnerVolumeSpecName "kube-api-access-p6jkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:26:37.094237 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:37.094151    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1642c6ef-7c39-41f8-86d4-d1f7780aca05-openshift-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:37.094237 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:37.094179    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-maas-api-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:37.094237 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:37.094192    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jkx\" (UniqueName: \"kubernetes.io/projected/1642c6ef-7c39-41f8-86d4-d1f7780aca05-kube-api-access-p6jkx\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:37.094237 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:37.094204    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1642c6ef-7c39-41f8-86d4-d1f7780aca05-metrics-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:37.161570 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:37.161534    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"]
Sep 11 01:26:37.165000 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:37.164972    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f6824-8bf778546-lc84r"]
Sep 11 01:26:39.036892 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:39.036852    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1642c6ef-7c39-41f8-86d4-d1f7780aca05" path="/var/lib/kubelet/pods/1642c6ef-7c39-41f8-86d4-d1f7780aca05/volumes"
Sep 11 01:26:39.483859 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:39.483828    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:26:43.235960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:43.235931    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"]
Sep 11 01:26:43.236417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:43.236144    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy" containerID="cri-o://acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8" gracePeriod=30
Sep 11 01:26:48.481116 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.481094    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:26:48.485367 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485347    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-certs\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485486 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485384    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-credential-socket\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485486 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485412    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-data\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485486 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485453    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-token\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485647 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485494    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istiod-ca-cert\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485647 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-envoy\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485647 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-podinfo\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485647 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485622    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:48.485647 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485635    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:48.485895 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485639    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-socket\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485895 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgl9t\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-kube-api-access-mgl9t\") pod \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\" (UID: \"af44596c-92c9-4b02-9db8-d2ee0b5b85af\") "
Sep 11 01:26:48.485895 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485745    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-data" (OuterVolumeSpecName: "istio-data") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:48.486052 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485906    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:48.486052 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.485943    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:26:48.486163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.486074    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.486163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.486092    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.486163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.486106    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.486163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.486119    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.486163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.486132    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.487716 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.487690    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-token" (OuterVolumeSpecName: "istio-token") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:26:48.487848 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.487730    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:26:48.488063 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.488044    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-kube-api-access-mgl9t" (OuterVolumeSpecName: "kube-api-access-mgl9t") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "kube-api-access-mgl9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:26:48.488131 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.488047    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af44596c-92c9-4b02-9db8-d2ee0b5b85af" (UID: "af44596c-92c9-4b02-9db8-d2ee0b5b85af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:26:48.586562 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.586531    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgl9t\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-kube-api-access-mgl9t\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.586562 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.586559    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.586741 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.586569    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.586741 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.586577    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af44596c-92c9-4b02-9db8-d2ee0b5b85af-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:26:48.907067 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.906976    2583 generic.go:356] "Generic (PLEG): container finished" podID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerID="acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8" exitCode=0
Sep 11 01:26:48.907067 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.907044    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"
Sep 11 01:26:48.907242 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.907066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" event={"ID":"af44596c-92c9-4b02-9db8-d2ee0b5b85af","Type":"ContainerDied","Data":"acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8"}
Sep 11 01:26:48.907242 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.907150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx" event={"ID":"af44596c-92c9-4b02-9db8-d2ee0b5b85af","Type":"ContainerDied","Data":"9bead5267f563bf06635599cafd2a63d5f0892110363d808d2876d54e11ca728"}
Sep 11 01:26:48.907242 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.907172    2583 scope.go:117] "RemoveContainer" containerID="acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8"
Sep 11 01:26:48.916189 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.916173    2583 scope.go:117] "RemoveContainer" containerID="e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77"
Sep 11 01:26:48.924489 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.924470    2583 scope.go:117] "RemoveContainer" containerID="acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8"
Sep 11 01:26:48.924799 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:26:48.924779    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8\": container with ID starting with acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8 not found: ID does not exist" containerID="acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8"
Sep 11 01:26:48.924884 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.924811    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8"} err="failed to get container status \"acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8\": rpc error: code = NotFound desc = could not find container \"acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8\": container with ID starting with acdf3d1eedfbc47c9b565ce957c2b278e300c728f8c45905d7f44bd927b054c8 not found: ID does not exist"
Sep 11 01:26:48.924884 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.924836    2583 scope.go:117] "RemoveContainer" containerID="e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77"
Sep 11 01:26:48.925116 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:26:48.925098    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77\": container with ID starting with e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77 not found: ID does not exist" containerID="e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77"
Sep 11 01:26:48.925173 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.925125    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77"} err="failed to get container status \"e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77\": rpc error: code = NotFound desc = could not find container \"e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77\": container with ID starting with e29e98ef53d8990c616f652bd536ed8796335994c97faf0e8473919621a82c77 not found: ID does not exist"
Sep 11 01:26:48.928512 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.928492    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"]
Sep 11 01:26:48.932597 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:48.932576    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f6824-openshift-default-fcb676cb4-d7lxx"]
Sep 11 01:26:49.036353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:49.036296    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" path="/var/lib/kubelet/pods/af44596c-92c9-4b02-9db8-d2ee0b5b85af/volumes"
Sep 11 01:26:56.677808 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.677768    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"]
Sep 11 01:26:56.678286 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678266    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy"
Sep 11 01:26:56.678390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678291    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy"
Sep 11 01:26:56.678390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678329    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f09932b8-f03a-4618-a119-8b69c5b48569" containerName="payload-pre-processing"
Sep 11 01:26:56.678390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678339    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f09932b8-f03a-4618-a119-8b69c5b48569" containerName="payload-pre-processing"
Sep 11 01:26:56.678390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678349    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy"
Sep 11 01:26:56.678390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678357    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy"
Sep 11 01:26:56.678390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678369    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cdab54-d747-4276-aced-bee307464f11" containerName="payload-processing"
Sep 11 01:26:56.678390 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678378    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cdab54-d747-4276-aced-bee307464f11" containerName="payload-processing"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678402    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678411    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678426    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" containerName="maas-api"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678434    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" containerName="maas-api"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678450    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678458    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678468    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1642c6ef-7c39-41f8-86d4-d1f7780aca05" containerName="maas-api"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678476    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1642c6ef-7c39-41f8-86d4-d1f7780aca05" containerName="maas-api"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678486    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678494    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678580    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cdab54-d747-4276-aced-bee307464f11" containerName="payload-processing"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678592    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678601    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678612    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678620    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678629    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f4ce473-ffb2-4a6d-b64a-bbf33e9e4f57" containerName="maas-api"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678638    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8339de5b-ae7f-4bb6-b917-9f0e0919f92e" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678647    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="af44596c-92c9-4b02-9db8-d2ee0b5b85af" containerName="istio-proxy"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678654    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1642c6ef-7c39-41f8-86d4-d1f7780aca05" containerName="maas-api"
Sep 11 01:26:56.678740 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.678665    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f09932b8-f03a-4618-a119-8b69c5b48569" containerName="payload-pre-processing"
Sep 11 01:26:56.680984 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.680962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.683353 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.683329    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b9930f\"/\"openshift-service-ca.crt\""
Sep 11 01:26:56.684140 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.684118    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b9930f\"/\"kube-root-ca.crt\""
Sep 11 01:26:56.684227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.684133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b9930f\"/\"default-dockercfg-tzbh8\""
Sep 11 01:26:56.684227 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.684168    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b9930f\"/\"sub-test-model-b9930f-kserve-self-signed-certs\""
Sep 11 01:26:56.690919 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.690894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"]
Sep 11 01:26:56.744811 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.744782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-dshm\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.744931 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.744833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-home\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.744931 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.744857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tls-certs\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.745016 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.744935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgmp\" (UniqueName: \"kubernetes.io/projected/4f1bec7e-5855-4ab6-a207-2df485b26ac7-kube-api-access-lzgmp\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.745016 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.744989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-model-cache\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.745016 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.745008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tmp-dir\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846170 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-model-cache\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846170 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tmp-dir\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-dshm\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-home\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846417 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tls-certs\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846577 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgmp\" (UniqueName: \"kubernetes.io/projected/4f1bec7e-5855-4ab6-a207-2df485b26ac7-kube-api-access-lzgmp\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846635 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846597    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-model-cache\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846704 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846679    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tmp-dir\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.846783 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.846764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-home\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.848420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.848401    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-dshm\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.848915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.848894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tls-certs\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.854558 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.854533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgmp\" (UniqueName: \"kubernetes.io/projected/4f1bec7e-5855-4ab6-a207-2df485b26ac7-kube-api-access-lzgmp\") pod \"sub-test-model-b9930f-kserve-7647bd6b69-b49gd\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:56.991419 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:56.991339    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:26:57.264018 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:57.263991    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"]
Sep 11 01:26:57.265821 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:26:57.265794    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f1bec7e_5855_4ab6_a207_2df485b26ac7.slice/crio-454eaf82ce2ffbeeb312ff155cd24abe1e3880e1c0523e37d96bf7901f17af7e WatchSource:0}: Error finding container 454eaf82ce2ffbeeb312ff155cd24abe1e3880e1c0523e37d96bf7901f17af7e: Status 404 returned error can't find the container with id 454eaf82ce2ffbeeb312ff155cd24abe1e3880e1c0523e37d96bf7901f17af7e
Sep 11 01:26:57.952088 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:57.952053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd" event={"ID":"4f1bec7e-5855-4ab6-a207-2df485b26ac7","Type":"ContainerStarted","Data":"af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1"}
Sep 11 01:26:57.952465 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:57.952099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd" event={"ID":"4f1bec7e-5855-4ab6-a207-2df485b26ac7","Type":"ContainerStarted","Data":"454eaf82ce2ffbeeb312ff155cd24abe1e3880e1c0523e37d96bf7901f17af7e"}
Sep 11 01:26:57.971387 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:26:57.971344    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd" podStartSLOduration=1.971329112 podStartE2EDuration="1.971329112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:26:57.969290959 +0000 UTC m=+1747.505942160" watchObservedRunningTime="2026-09-11 01:26:57.971329112 +0000 UTC m=+1747.507980549"
Sep 11 01:27:00.277641 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:00.277598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:27:06.992127 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:06.992091    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:27:06.992647 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:06.992242    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:27:07.004821 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:07.004790    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:27:08.008686 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:08.008658    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:27:38.875071 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:38.875031    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:27:49.915606 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:49.915572    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7965-be9382"]
Sep 11 01:27:49.916007 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:49.915903    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy"
Sep 11 01:27:49.916007 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:49.915915    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9592c11a-0202-47a8-8f48-e55ae5978e8f" containerName="istio-proxy"
Sep 11 01:27:49.918764 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:49.918745    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:49.921013 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:49.920990    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7gdtv\""
Sep 11 01:27:49.925118 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:49.925097    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7965-be9382"]
Sep 11 01:27:50.010904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:50.010864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q5bw\" (UniqueName: \"kubernetes.io/projected/fa82498b-7d11-4dd2-8b1c-5b657e0ff106-kube-api-access-4q5bw\") pod \"test-curl-7965-be9382\" (UID: \"fa82498b-7d11-4dd2-8b1c-5b657e0ff106\") " pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:50.111472 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:50.111436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q5bw\" (UniqueName: \"kubernetes.io/projected/fa82498b-7d11-4dd2-8b1c-5b657e0ff106-kube-api-access-4q5bw\") pod \"test-curl-7965-be9382\" (UID: \"fa82498b-7d11-4dd2-8b1c-5b657e0ff106\") " pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:50.120076 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:50.120041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q5bw\" (UniqueName: \"kubernetes.io/projected/fa82498b-7d11-4dd2-8b1c-5b657e0ff106-kube-api-access-4q5bw\") pod \"test-curl-7965-be9382\" (UID: \"fa82498b-7d11-4dd2-8b1c-5b657e0ff106\") " pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:50.231950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:50.231873    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:50.502876 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:50.502798    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7965-be9382"]
Sep 11 01:27:50.505147 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:27:50.505116    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa82498b_7d11_4dd2_8b1c_5b657e0ff106.slice/crio-2770b15245dcccb649c167e8c5c8b57ba9d45001a7769d237622e9d10bde61c7 WatchSource:0}: Error finding container 2770b15245dcccb649c167e8c5c8b57ba9d45001a7769d237622e9d10bde61c7: Status 404 returned error can't find the container with id 2770b15245dcccb649c167e8c5c8b57ba9d45001a7769d237622e9d10bde61c7
Sep 11 01:27:50.999867 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:50.999838    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:27:51.003655 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:51.003630    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:27:51.005192 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:51.005163    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:27:51.008901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:51.008884    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:27:51.201634 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:51.201596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7965-be9382" event={"ID":"fa82498b-7d11-4dd2-8b1c-5b657e0ff106","Type":"ContainerStarted","Data":"6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc"}
Sep 11 01:27:51.201634 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:51.201638    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7965-be9382" event={"ID":"fa82498b-7d11-4dd2-8b1c-5b657e0ff106","Type":"ContainerStarted","Data":"2770b15245dcccb649c167e8c5c8b57ba9d45001a7769d237622e9d10bde61c7"}
Sep 11 01:27:51.215007 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:51.214954    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7965-be9382" podStartSLOduration=2.214936594 podStartE2EDuration="2.214936594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:27:51.214782378 +0000 UTC m=+1800.751433581" watchObservedRunningTime="2026-09-11 01:27:51.214936594 +0000 UTC m=+1800.751587795"
Sep 11 01:27:51.828262 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:51.828230    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7965-be9382"]
Sep 11 01:27:51.832359 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:51.832329    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7965-be9382"]
Sep 11 01:27:52.206481 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:27:52.206399    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7965-be9382" container="test-curl-7965-be9382"
Sep 11 01:27:53.211030 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:53.210986    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7965-be9382" podUID="fa82498b-7d11-4dd2-8b1c-5b657e0ff106" containerName="test-curl-7965-be9382" containerID="cri-o://6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc" gracePeriod=2
Sep 11 01:27:54.077129 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:54.077097    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:27:55.393086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:55.393056    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:55.459377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:55.459339    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q5bw\" (UniqueName: \"kubernetes.io/projected/fa82498b-7d11-4dd2-8b1c-5b657e0ff106-kube-api-access-4q5bw\") pod \"fa82498b-7d11-4dd2-8b1c-5b657e0ff106\" (UID: \"fa82498b-7d11-4dd2-8b1c-5b657e0ff106\") "
Sep 11 01:27:55.461280 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:55.461254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa82498b-7d11-4dd2-8b1c-5b657e0ff106-kube-api-access-4q5bw" (OuterVolumeSpecName: "kube-api-access-4q5bw") pod "fa82498b-7d11-4dd2-8b1c-5b657e0ff106" (UID: "fa82498b-7d11-4dd2-8b1c-5b657e0ff106"). InnerVolumeSpecName "kube-api-access-4q5bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:27:55.560848 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:55.560818    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q5bw\" (UniqueName: \"kubernetes.io/projected/fa82498b-7d11-4dd2-8b1c-5b657e0ff106-kube-api-access-4q5bw\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:27:56.228071 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:56.228036    2583 generic.go:356] "Generic (PLEG): container finished" podID="fa82498b-7d11-4dd2-8b1c-5b657e0ff106" containerID="6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc" exitCode=137
Sep 11 01:27:56.228266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:56.228091    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:56.228266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:56.228125    2583 scope.go:117] "RemoveContainer" containerID="6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc"
Sep 11 01:27:56.238511 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:56.238488    2583 scope.go:117] "RemoveContainer" containerID="6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc"
Sep 11 01:27:56.238639 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:27:56.238584    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7965-be9382\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="fa82498b-7d11-4dd2-8b1c-5b657e0ff106" pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:56.238818 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:27:56.238800    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc\": container with ID starting with 6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc not found: ID does not exist" containerID="6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc"
Sep 11 01:27:56.238882 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:56.238828    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc"} err="failed to get container status \"6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc\": rpc error: code = NotFound desc = could not find container \"6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc\": container with ID starting with 6181d5b92c68e2a24586eac3982ec1c0fe9dbdde22989a38a76bc5862a4054bc not found: ID does not exist"
Sep 11 01:27:56.689773 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:56.689732    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"]
Sep 11 01:27:56.691888 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:27:56.691858    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7965-be9382\" is forbidden: User \"system:node:ip-10-0-142-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-82.ec2.internal' and this object" podUID="fa82498b-7d11-4dd2-8b1c-5b657e0ff106" pod="opendatahub/test-curl-7965-be9382"
Sep 11 01:27:57.037189 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:27:57.037113    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa82498b-7d11-4dd2-8b1c-5b657e0ff106" path="/var/lib/kubelet/pods/fa82498b-7d11-4dd2-8b1c-5b657e0ff106/volumes"
Sep 11 01:28:05.378243 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:05.378208    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:28:11.708488 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:11.708446    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd" podUID="4f1bec7e-5855-4ab6-a207-2df485b26ac7" containerName="main" containerID="cri-o://af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1" gracePeriod=45
Sep 11 01:28:11.962551 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:11.962497    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:28:12.114214 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114185    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-dshm\") pod \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") "
Sep 11 01:28:12.114214 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114216    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-home\") pod \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") "
Sep 11 01:28:12.114477 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114236    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzgmp\" (UniqueName: \"kubernetes.io/projected/4f1bec7e-5855-4ab6-a207-2df485b26ac7-kube-api-access-lzgmp\") pod \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") "
Sep 11 01:28:12.114477 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114264    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tls-certs\") pod \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") "
Sep 11 01:28:12.114477 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-model-cache\") pod \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") "
Sep 11 01:28:12.114477 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tmp-dir\") pod \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\" (UID: \"4f1bec7e-5855-4ab6-a207-2df485b26ac7\") "
Sep 11 01:28:12.114685 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114527    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-home" (OuterVolumeSpecName: "home") pod "4f1bec7e-5855-4ab6-a207-2df485b26ac7" (UID: "4f1bec7e-5855-4ab6-a207-2df485b26ac7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:28:12.114685 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114605    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f1bec7e-5855-4ab6-a207-2df485b26ac7" (UID: "4f1bec7e-5855-4ab6-a207-2df485b26ac7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:28:12.114783 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114764    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f1bec7e-5855-4ab6-a207-2df485b26ac7" (UID: "4f1bec7e-5855-4ab6-a207-2df485b26ac7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:28:12.114837 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114782    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-model-cache\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:28:12.114837 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.114801    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-home\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:28:12.116537 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.116510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-dshm" (OuterVolumeSpecName: "dshm") pod "4f1bec7e-5855-4ab6-a207-2df485b26ac7" (UID: "4f1bec7e-5855-4ab6-a207-2df485b26ac7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:28:12.116661 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.116581    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f1bec7e-5855-4ab6-a207-2df485b26ac7" (UID: "4f1bec7e-5855-4ab6-a207-2df485b26ac7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:28:12.116661 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.116585    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1bec7e-5855-4ab6-a207-2df485b26ac7-kube-api-access-lzgmp" (OuterVolumeSpecName: "kube-api-access-lzgmp") pod "4f1bec7e-5855-4ab6-a207-2df485b26ac7" (UID: "4f1bec7e-5855-4ab6-a207-2df485b26ac7"). InnerVolumeSpecName "kube-api-access-lzgmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:28:12.219994 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.216527    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tmp-dir\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:28:12.219994 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.216562    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bec7e-5855-4ab6-a207-2df485b26ac7-dshm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:28:12.219994 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.216586    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgmp\" (UniqueName: \"kubernetes.io/projected/4f1bec7e-5855-4ab6-a207-2df485b26ac7-kube-api-access-lzgmp\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:28:12.219994 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.216609    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bec7e-5855-4ab6-a207-2df485b26ac7-tls-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:28:12.311726 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.311693    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f1bec7e-5855-4ab6-a207-2df485b26ac7" containerID="af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1" exitCode=0
Sep 11 01:28:12.311904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.311755    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"
Sep 11 01:28:12.311904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.311772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd" event={"ID":"4f1bec7e-5855-4ab6-a207-2df485b26ac7","Type":"ContainerDied","Data":"af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1"}
Sep 11 01:28:12.311904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.311813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd" event={"ID":"4f1bec7e-5855-4ab6-a207-2df485b26ac7","Type":"ContainerDied","Data":"454eaf82ce2ffbeeb312ff155cd24abe1e3880e1c0523e37d96bf7901f17af7e"}
Sep 11 01:28:12.311904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.311830    2583 scope.go:117] "RemoveContainer" containerID="af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1"
Sep 11 01:28:12.322252 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.322235    2583 scope.go:117] "RemoveContainer" containerID="af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1"
Sep 11 01:28:12.322580 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:28:12.322558    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1\": container with ID starting with af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1 not found: ID does not exist" containerID="af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1"
Sep 11 01:28:12.322652 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.322590    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1"} err="failed to get container status \"af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1\": rpc error: code = NotFound desc = could not find container \"af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1\": container with ID starting with af596937c276b29f8afb24e160a8a82f578144177e80c9cbe95f099e19e559d1 not found: ID does not exist"
Sep 11 01:28:12.335640 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.335619    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"]
Sep 11 01:28:12.337187 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:12.337168    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/sub-test-model-b9930f-kserve-7647bd6b69-b49gd"]
Sep 11 01:28:13.036264 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:13.036230    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f1bec7e-5855-4ab6-a207-2df485b26ac7" path="/var/lib/kubelet/pods/4f1bec7e-5855-4ab6-a207-2df485b26ac7/volumes"
Sep 11 01:28:26.982384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.982351    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"]
Sep 11 01:28:26.982901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.982767    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1bec7e-5855-4ab6-a207-2df485b26ac7" containerName="main"
Sep 11 01:28:26.982901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.982782    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1bec7e-5855-4ab6-a207-2df485b26ac7" containerName="main"
Sep 11 01:28:26.982901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.982797    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa82498b-7d11-4dd2-8b1c-5b657e0ff106" containerName="test-curl-7965-be9382"
Sep 11 01:28:26.982901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.982802    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa82498b-7d11-4dd2-8b1c-5b657e0ff106" containerName="test-curl-7965-be9382"
Sep 11 01:28:26.982901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.982865    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa82498b-7d11-4dd2-8b1c-5b657e0ff106" containerName="test-curl-7965-be9382"
Sep 11 01:28:26.982901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.982876    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f1bec7e-5855-4ab6-a207-2df485b26ac7" containerName="main"
Sep 11 01:28:26.985829 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.985812    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:26.988124 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.988098    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b9930f\"/\"openshift-service-ca.crt\""
Sep 11 01:28:26.988901 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.988884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b9930f\"/\"rate-test-model-b9930f-kserve-self-signed-certs\""
Sep 11 01:28:26.989036 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.989012    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b9930f\"/\"default-dockercfg-tzbh8\""
Sep 11 01:28:26.989105 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.989046    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b9930f\"/\"kube-root-ca.crt\""
Sep 11 01:28:26.995923 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:26.995902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"]
Sep 11 01:28:27.039200 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.039173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj4kx\" (UniqueName: \"kubernetes.io/projected/a59b5520-f190-4cb6-914a-ef6dfe952ced-kube-api-access-zj4kx\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.039370 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.039210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59b5520-f190-4cb6-914a-ef6dfe952ced-tls-certs\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.039370 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.039238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-dshm\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.039370 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.039287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-model-cache\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.039488 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.039379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-home\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.039488 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.039416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-tmp-dir\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.140706 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.140670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4kx\" (UniqueName: \"kubernetes.io/projected/a59b5520-f190-4cb6-914a-ef6dfe952ced-kube-api-access-zj4kx\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.140865 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.140716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59b5520-f190-4cb6-914a-ef6dfe952ced-tls-certs\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.140865 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.140748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-dshm\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.140865 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.140762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-model-cache\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.140865 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.140782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-home\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.140865 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.140805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-tmp-dir\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.141163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.141141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-tmp-dir\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.141247 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.141226    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-home\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.141247 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.141243    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-model-cache\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.143044 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.143023    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-dshm\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.143377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.143357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59b5520-f190-4cb6-914a-ef6dfe952ced-tls-certs\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.147956 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.147937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4kx\" (UniqueName: \"kubernetes.io/projected/a59b5520-f190-4cb6-914a-ef6dfe952ced-kube-api-access-zj4kx\") pod \"rate-test-model-b9930f-kserve-5cc8d48475-nlcdx\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") " pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.296180 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.296102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:27.569640 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:27.569617    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"]
Sep 11 01:28:27.571495 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:28:27.571460    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda59b5520_f190_4cb6_914a_ef6dfe952ced.slice/crio-b0843e3493f720d8156b774bc320c9c450e0a1058114622c4289ca2d1d6ad87a WatchSource:0}: Error finding container b0843e3493f720d8156b774bc320c9c450e0a1058114622c4289ca2d1d6ad87a: Status 404 returned error can't find the container with id b0843e3493f720d8156b774bc320c9c450e0a1058114622c4289ca2d1d6ad87a
Sep 11 01:28:28.389029 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:28.388996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx" event={"ID":"a59b5520-f190-4cb6-914a-ef6dfe952ced","Type":"ContainerStarted","Data":"eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615"}
Sep 11 01:28:28.389428 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:28.389037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx" event={"ID":"a59b5520-f190-4cb6-914a-ef6dfe952ced","Type":"ContainerStarted","Data":"b0843e3493f720d8156b774bc320c9c450e0a1058114622c4289ca2d1d6ad87a"}
Sep 11 01:28:28.404697 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:28.404645    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx" podStartSLOduration=2.404627264 podStartE2EDuration="2.404627264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:28:28.404432064 +0000 UTC m=+1837.941083265" watchObservedRunningTime="2026-09-11 01:28:28.404627264 +0000 UTC m=+1837.941278467"
Sep 11 01:28:30.876338 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:30.876285    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:28:37.296627 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:37.296582    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:37.297146 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:37.296658    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:37.309249 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:37.309226    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:37.444011 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:37.443983    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:28:45.371693 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:45.371653    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:28:54.874773 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:28:54.874732    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:29:06.876253 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:06.876215    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:29:12.164106 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:12.164071    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"]
Sep 11 01:29:12.914063 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:12.914029    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"]
Sep 11 01:29:12.917400 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:12.917379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:12.919821 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:12.919800    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 01:29:12.920035 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:12.920015    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c1edc6-openshift-default-dockercfg-mqk85\""
Sep 11 01:29:12.928651 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:12.928629    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"]
Sep 11 01:29:13.039858 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.039830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-token\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.039858 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.039862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-data\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.040098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.039880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-socket\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.040098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.039950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-credential-socket\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.040098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.040010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njq7\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-kube-api-access-8njq7\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.040098 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.040046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-podinfo\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.040236 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.040147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-envoy\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.040236 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.040190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-certs\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.040301 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.040238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istiod-ca-cert\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.140803 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.140772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-certs\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.140950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.140808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istiod-ca-cert\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.140950 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.140932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-token\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141063 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.140963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-data\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141063 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.140986    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-socket\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141063 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-credential-socket\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141213 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njq7\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-kube-api-access-8njq7\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141213 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-podinfo\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141213 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-envoy\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141213 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-certs\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141454 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141433    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-data\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141546 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-credential-socket\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141609 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-socket\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.141609 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.141568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istiod-ca-cert\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.143566 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.143547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-envoy\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.143698 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.143682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-podinfo\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.149084 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.149061    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-token\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.149302 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.149287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njq7\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-kube-api-access-8njq7\") pod \"e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") " pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.230644 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.230541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:13.506193 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.506169    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"]
Sep 11 01:29:13.507935 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:29:13.507902    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef7c2c7_9857_4238_94a5_bb1ed0d5420e.slice/crio-4ac2c255942723f9e64ddd55d41041d15b58ec72320c0b84a96486099e480cd2 WatchSource:0}: Error finding container 4ac2c255942723f9e64ddd55d41041d15b58ec72320c0b84a96486099e480cd2: Status 404 returned error can't find the container with id 4ac2c255942723f9e64ddd55d41041d15b58ec72320c0b84a96486099e480cd2
Sep 11 01:29:13.510254 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.510226    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:13.510373 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.510287    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:13.510373 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.510352    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:13.607359 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.607325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" event={"ID":"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e","Type":"ContainerStarted","Data":"3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1"}
Sep 11 01:29:13.607506 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.607386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" event={"ID":"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e","Type":"ContainerStarted","Data":"4ac2c255942723f9e64ddd55d41041d15b58ec72320c0b84a96486099e480cd2"}
Sep 11 01:29:13.625836 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:13.625779    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podStartSLOduration=1.625756612 podStartE2EDuration="1.625756612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:29:13.624840414 +0000 UTC m=+1883.161491638" watchObservedRunningTime="2026-09-11 01:29:13.625756612 +0000 UTC m=+1883.162407815"
Sep 11 01:29:14.231163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:14.231126    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:14.236722 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:14.236698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:14.612303 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:14.612273    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:14.619010 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:14.618981    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:16.581630 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.581598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:29:16.677991 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.677960    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"]
Sep 11 01:29:16.681842 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.681820    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.685136 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.685109    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c1edc6-serving-cert\""
Sep 11 01:29:16.685136 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.685130    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 01:29:16.685349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.685118    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fdvhg\""
Sep 11 01:29:16.685349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.685188    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:29:16.685349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.685160    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 01:29:16.693942 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.693916    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"]
Sep 11 01:29:16.876083 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.875998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.876083 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.876050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-metrics-certs\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.876288 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.876139    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk8s5\" (UniqueName: \"kubernetes.io/projected/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-kube-api-access-tk8s5\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.876288 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.876206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.977130 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.977085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.977350 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.977138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-metrics-certs\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.977350 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.977187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk8s5\" (UniqueName: \"kubernetes.io/projected/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-kube-api-access-tk8s5\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.977350 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.977247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.977479 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:29:16.977378    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-serving-cert: secret "maas-api-e2e-ipp-a-c1edc6-serving-cert" not found
Sep 11 01:29:16.977479 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:29:16.977444    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls podName:0cd14cd2-b3a5-4169-ba5e-81bcd93912a0 nodeName:}" failed. No retries permitted until 2026-09-11 01:29:17.477421528 +0000 UTC m=+1887.014072712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls") pod "maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x" (UID: "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0") : secret "maas-api-e2e-ipp-a-c1edc6-serving-cert" not found
Sep 11 01:29:16.977805 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.977775    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.979718 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.979695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-metrics-certs\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:16.985356 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:16.985336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk8s5\" (UniqueName: \"kubernetes.io/projected/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-kube-api-access-tk8s5\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:17.482747 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:17.482714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:17.485194 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:17.485168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:17.593524 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:17.593492    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:17.868377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:17.868349    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"]
Sep 11 01:29:17.870239 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:29:17.870212    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd14cd2_b3a5_4169_ba5e_81bcd93912a0.slice/crio-c41c2f253f3f87d0e00c987a6cedf7d622d83b11994ee87afcaf24ba26ecb0f8 WatchSource:0}: Error finding container c41c2f253f3f87d0e00c987a6cedf7d622d83b11994ee87afcaf24ba26ecb0f8: Status 404 returned error can't find the container with id c41c2f253f3f87d0e00c987a6cedf7d622d83b11994ee87afcaf24ba26ecb0f8
Sep 11 01:29:18.633875 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:18.633842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x" event={"ID":"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0","Type":"ContainerStarted","Data":"c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271"}
Sep 11 01:29:18.634270 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:18.633889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x" event={"ID":"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0","Type":"ContainerStarted","Data":"c41c2f253f3f87d0e00c987a6cedf7d622d83b11994ee87afcaf24ba26ecb0f8"}
Sep 11 01:29:18.634270 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:18.633965    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:18.652462 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:18.652420    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x" podStartSLOduration=2.254285227 podStartE2EDuration="2.652403972s" totalImagesPullingTime="398.118745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:29:18.650129493 +0000 UTC m=+1888.186780867" watchObservedRunningTime="2026-09-11 01:29:18.652403972 +0000 UTC m=+1888.189055173"
Sep 11 01:29:24.648366 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:24.648335    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:29:27.184160 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.184106    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx" podUID="a59b5520-f190-4cb6-914a-ef6dfe952ced" containerName="main" containerID="cri-o://eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615" gracePeriod=45
Sep 11 01:29:27.475421 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.475399    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:29:27.575187 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59b5520-f190-4cb6-914a-ef6dfe952ced-tls-certs\") pod \"a59b5520-f190-4cb6-914a-ef6dfe952ced\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") "
Sep 11 01:29:27.575378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575196    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-model-cache\") pod \"a59b5520-f190-4cb6-914a-ef6dfe952ced\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") "
Sep 11 01:29:27.575378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-dshm\") pod \"a59b5520-f190-4cb6-914a-ef6dfe952ced\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") "
Sep 11 01:29:27.575378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-tmp-dir\") pod \"a59b5520-f190-4cb6-914a-ef6dfe952ced\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") "
Sep 11 01:29:27.575378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575281    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-home\") pod \"a59b5520-f190-4cb6-914a-ef6dfe952ced\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") "
Sep 11 01:29:27.575378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575331    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj4kx\" (UniqueName: \"kubernetes.io/projected/a59b5520-f190-4cb6-914a-ef6dfe952ced-kube-api-access-zj4kx\") pod \"a59b5520-f190-4cb6-914a-ef6dfe952ced\" (UID: \"a59b5520-f190-4cb6-914a-ef6dfe952ced\") "
Sep 11 01:29:27.575665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575584    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a59b5520-f190-4cb6-914a-ef6dfe952ced" (UID: "a59b5520-f190-4cb6-914a-ef6dfe952ced"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:27.575665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575615    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-home" (OuterVolumeSpecName: "home") pod "a59b5520-f190-4cb6-914a-ef6dfe952ced" (UID: "a59b5520-f190-4cb6-914a-ef6dfe952ced"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:27.575665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.575637    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-model-cache" (OuterVolumeSpecName: "model-cache") pod "a59b5520-f190-4cb6-914a-ef6dfe952ced" (UID: "a59b5520-f190-4cb6-914a-ef6dfe952ced"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:27.577610 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.577573    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-dshm" (OuterVolumeSpecName: "dshm") pod "a59b5520-f190-4cb6-914a-ef6dfe952ced" (UID: "a59b5520-f190-4cb6-914a-ef6dfe952ced"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:29:27.577741 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.577670    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59b5520-f190-4cb6-914a-ef6dfe952ced-kube-api-access-zj4kx" (OuterVolumeSpecName: "kube-api-access-zj4kx") pod "a59b5520-f190-4cb6-914a-ef6dfe952ced" (UID: "a59b5520-f190-4cb6-914a-ef6dfe952ced"). InnerVolumeSpecName "kube-api-access-zj4kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:29:27.577741 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.577726    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59b5520-f190-4cb6-914a-ef6dfe952ced-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a59b5520-f190-4cb6-914a-ef6dfe952ced" (UID: "a59b5520-f190-4cb6-914a-ef6dfe952ced"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:29:27.676319 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.676276    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59b5520-f190-4cb6-914a-ef6dfe952ced-tls-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:29:27.676319 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.676303    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-model-cache\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:29:27.676319 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.676324    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-dshm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:29:27.676561 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.676331    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-tmp-dir\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:29:27.676561 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.676339    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59b5520-f190-4cb6-914a-ef6dfe952ced-home\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:29:27.676561 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.676348    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj4kx\" (UniqueName: \"kubernetes.io/projected/a59b5520-f190-4cb6-914a-ef6dfe952ced-kube-api-access-zj4kx\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:29:27.680219 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.680191    2583 generic.go:356] "Generic (PLEG): container finished" podID="a59b5520-f190-4cb6-914a-ef6dfe952ced" containerID="eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615" exitCode=0
Sep 11 01:29:27.680366 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.680237    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"
Sep 11 01:29:27.680366 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.680269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx" event={"ID":"a59b5520-f190-4cb6-914a-ef6dfe952ced","Type":"ContainerDied","Data":"eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615"}
Sep 11 01:29:27.680366 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.680345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx" event={"ID":"a59b5520-f190-4cb6-914a-ef6dfe952ced","Type":"ContainerDied","Data":"b0843e3493f720d8156b774bc320c9c450e0a1058114622c4289ca2d1d6ad87a"}
Sep 11 01:29:27.680366 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.680363    2583 scope.go:117] "RemoveContainer" containerID="eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615"
Sep 11 01:29:27.689245 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.689230    2583 scope.go:117] "RemoveContainer" containerID="eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615"
Sep 11 01:29:27.689541 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:29:27.689521    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615\": container with ID starting with eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615 not found: ID does not exist" containerID="eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615"
Sep 11 01:29:27.689597 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.689554    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615"} err="failed to get container status \"eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615\": rpc error: code = NotFound desc = could not find container \"eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615\": container with ID starting with eb420fd5bfd423bc95856b012d2c3c312d83c3a75ab86ec89b8771ae45ff1615 not found: ID does not exist"
Sep 11 01:29:27.700910 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.700889    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"]
Sep 11 01:29:27.706573 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:27.706528    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx"]
Sep 11 01:29:29.042152 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:29.042116    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a59b5520-f190-4cb6-914a-ef6dfe952ced" path="/var/lib/kubelet/pods/a59b5520-f190-4cb6-914a-ef6dfe952ced/volumes"
Sep 11 01:29:29.082070 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:29.082035    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:29:30.653511 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:30.653466    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 01:29:30.653920 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:30.653535    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 01:29:32.434130 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:32.434092    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-b9930f/rate-test-model-b9930f-kserve-5cc8d48475-nlcdx" podUID="a59b5520-f190-4cb6-914a-ef6dfe952ced" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 01:29:32.710212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:32.710122    2583 generic.go:356] "Generic (PLEG): container finished" podID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerID="3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1" exitCode=0
Sep 11 01:29:32.710212 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:32.710192    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" event={"ID":"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e","Type":"ContainerDied","Data":"3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1"}
Sep 11 01:29:32.710618 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:32.710599    2583 scope.go:117] "RemoveContainer" containerID="3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1"
Sep 11 01:29:32.711591 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:32.711569    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:32.711654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:32.711632    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:32.711696 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:32.711663    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:33.716974 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:33.716937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" event={"ID":"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e","Type":"ContainerStarted","Data":"76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630"}
Sep 11 01:29:34.231605 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:34.231551    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:34.232042 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:34.232012    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 01:29:34.232142 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:34.232081    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 01:29:35.231819 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:35.231775    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 01:29:35.232299 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:35.231872    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 01:29:36.231742 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:36.231699    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 01:29:36.231956 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:36.231768    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 01:29:37.232013 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:37.231959    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 01:29:37.232629 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:37.232585    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 01:29:38.231551 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:38.231509    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 01:29:38.231757 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:38.231574    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 01:29:39.234808 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:39.234771    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:39.235232 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:39.235134    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:39.235765 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:39.235745    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:29:41.599323 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.599272    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"]
Sep 11 01:29:41.599682 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.599660    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a59b5520-f190-4cb6-914a-ef6dfe952ced" containerName="main"
Sep 11 01:29:41.599682 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.599671    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a59b5520-f190-4cb6-914a-ef6dfe952ced" containerName="main"
Sep 11 01:29:41.599772 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.599762    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a59b5520-f190-4cb6-914a-ef6dfe952ced" containerName="main"
Sep 11 01:29:41.602941 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.602919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.605630 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.605607    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-40f90a-openshift-default-dockercfg-g9j5w\""
Sep 11 01:29:41.615167 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.615143    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"]
Sep 11 01:29:41.706656 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-credential-socket\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.706811 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-token\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.706811 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-data\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.706811 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2cc\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-kube-api-access-cv2cc\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.706922 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-certs\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.706922 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-podinfo\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.706922 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706893    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-envoy\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.707023 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istiod-ca-cert\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.707023 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.706969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-socket\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.807803 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.807762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-credential-socket\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808003 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.807834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-token\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808003 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.807871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-data\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808003 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.807939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2cc\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-kube-api-access-cv2cc\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808003 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.807976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-certs\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.808017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-podinfo\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.808042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-envoy\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.808090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istiod-ca-cert\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.808115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-socket\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.808192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-credential-socket\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808532 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.808497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-socket\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808671 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.808647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-certs\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.808744 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.808678    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-data\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.809063 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.809038    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istiod-ca-cert\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.810879 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.810852    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-envoy\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.810992 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.810974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-podinfo\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.815646 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.815622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-token\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.815917 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.815900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2cc\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-kube-api-access-cv2cc\") pod \"e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") " pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:41.915666 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:41.915574    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:42.194261 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:42.194233    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"]
Sep 11 01:29:42.196899 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:29:42.196871    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421d2f92_ddcf_4b84_ba3e_fc605947f26c.slice/crio-4eab6fccb4df3479e175f27f1c25f92e6d2a22aa8fc8a60fcf1ed930138a43c3 WatchSource:0}: Error finding container 4eab6fccb4df3479e175f27f1c25f92e6d2a22aa8fc8a60fcf1ed930138a43c3: Status 404 returned error can't find the container with id 4eab6fccb4df3479e175f27f1c25f92e6d2a22aa8fc8a60fcf1ed930138a43c3
Sep 11 01:29:42.199739 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:42.199712    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:42.199816 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:42.199772    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:42.199816 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:42.199797    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:29:42.770733 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:42.770696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" event={"ID":"421d2f92-ddcf-4b84-ba3e-fc605947f26c","Type":"ContainerStarted","Data":"817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780"}
Sep 11 01:29:42.771170 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:42.770747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" event={"ID":"421d2f92-ddcf-4b84-ba3e-fc605947f26c","Type":"ContainerStarted","Data":"4eab6fccb4df3479e175f27f1c25f92e6d2a22aa8fc8a60fcf1ed930138a43c3"}
Sep 11 01:29:42.797116 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:42.797070    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podStartSLOduration=1.797054887 podStartE2EDuration="1.797054887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:29:42.796106456 +0000 UTC m=+1912.332757657" watchObservedRunningTime="2026-09-11 01:29:42.797054887 +0000 UTC m=+1912.333706084"
Sep 11 01:29:42.916401 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:42.916366    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:43.921410 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:43.921380    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:44.779916 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:44.779877    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:44.780711 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:44.780688    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:29:52.085586 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:29:52.085546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:30:00.149716 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.149675    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"]
Sep 11 01:30:00.153680 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.153656    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"]
Sep 11 01:30:00.153834 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.153775    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"
Sep 11 01:30:00.156669 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.156641    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q6gbn\""
Sep 11 01:30:00.157825 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.157806    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"
Sep 11 01:30:00.165359 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.165335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"
Sep 11 01:30:00.168715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.168692    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"]
Sep 11 01:30:00.174373 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.174349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"
Sep 11 01:30:00.175267 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.175241    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"]
Sep 11 01:30:00.468644 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.468618    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"]
Sep 11 01:30:00.471005 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:30:00.470972    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c7696b_a8a7_4fdc_9d6f_1faa6f11fe18.slice/crio-65ebbacfb25b59f5928c6ef3d3247f925bdea13e4976d444320d4962ff159253 WatchSource:0}: Error finding container 65ebbacfb25b59f5928c6ef3d3247f925bdea13e4976d444320d4962ff159253: Status 404 returned error can't find the container with id 65ebbacfb25b59f5928c6ef3d3247f925bdea13e4976d444320d4962ff159253
Sep 11 01:30:00.474768 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.474743    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"]
Sep 11 01:30:00.477870 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:30:00.477847    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3419d95_37f7_4d46_a5fd_41327d44acc4.slice/crio-264b266245d63b18fed16845328f2513f58d606c29ef2ba98ac2c0a58e8dac0b WatchSource:0}: Error finding container 264b266245d63b18fed16845328f2513f58d606c29ef2ba98ac2c0a58e8dac0b: Status 404 returned error can't find the container with id 264b266245d63b18fed16845328f2513f58d606c29ef2ba98ac2c0a58e8dac0b
Sep 11 01:30:00.821515 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.821472    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 01:30:00.821718 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.821533    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 01:30:00.878466 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.878422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5" event={"ID":"43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18","Type":"ContainerStarted","Data":"7d79f060f9b8c642788afd5aa1b7db3b3de5d5c3d590275ba45072904117ba9a"}
Sep 11 01:30:00.878665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.878494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5" event={"ID":"43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18","Type":"ContainerStarted","Data":"65ebbacfb25b59f5928c6ef3d3247f925bdea13e4976d444320d4962ff159253"}
Sep 11 01:30:00.880026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.880002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv" event={"ID":"d3419d95-37f7-4d46-a5fd-41327d44acc4","Type":"ContainerStarted","Data":"f2f394191ff742b90517be3ffef8cf14ba949dcbeaad6a5cdb625a25f54e2c61"}
Sep 11 01:30:00.880144 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.880048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv" event={"ID":"d3419d95-37f7-4d46-a5fd-41327d44acc4","Type":"ContainerStarted","Data":"264b266245d63b18fed16845328f2513f58d606c29ef2ba98ac2c0a58e8dac0b"}
Sep 11 01:30:00.892972 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.892894    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5" podStartSLOduration=0.892872573 podStartE2EDuration="892.872573ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:30:00.891408183 +0000 UTC m=+1930.428059382" watchObservedRunningTime="2026-09-11 01:30:00.892872573 +0000 UTC m=+1930.429523775"
Sep 11 01:30:00.914247 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:00.914193    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv" podStartSLOduration=0.91417719 podStartE2EDuration="914.17719ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:30:00.912343258 +0000 UTC m=+1930.448994460" watchObservedRunningTime="2026-09-11 01:30:00.91417719 +0000 UTC m=+1930.450828391"
Sep 11 01:30:01.887548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:01.887505    2583 generic.go:356] "Generic (PLEG): container finished" podID="43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18" containerID="7d79f060f9b8c642788afd5aa1b7db3b3de5d5c3d590275ba45072904117ba9a" exitCode=0
Sep 11 01:30:01.887982 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:01.887589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5" event={"ID":"43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18","Type":"ContainerDied","Data":"7d79f060f9b8c642788afd5aa1b7db3b3de5d5c3d590275ba45072904117ba9a"}
Sep 11 01:30:01.889077 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:01.889051    2583 generic.go:356] "Generic (PLEG): container finished" podID="d3419d95-37f7-4d46-a5fd-41327d44acc4" containerID="f2f394191ff742b90517be3ffef8cf14ba949dcbeaad6a5cdb625a25f54e2c61" exitCode=0
Sep 11 01:30:01.889191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:01.889142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv" event={"ID":"d3419d95-37f7-4d46-a5fd-41327d44acc4","Type":"ContainerDied","Data":"f2f394191ff742b90517be3ffef8cf14ba949dcbeaad6a5cdb625a25f54e2c61"}
Sep 11 01:30:02.896603 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:02.896571    2583 generic.go:356] "Generic (PLEG): container finished" podID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerID="817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780" exitCode=0
Sep 11 01:30:02.897079 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:02.896647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" event={"ID":"421d2f92-ddcf-4b84-ba3e-fc605947f26c","Type":"ContainerDied","Data":"817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780"}
Sep 11 01:30:02.897631 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:02.897218    2583 scope.go:117] "RemoveContainer" containerID="817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780"
Sep 11 01:30:02.898468 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:02.898436    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:30:02.898568 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:02.898512    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:30:02.898568 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:02.898545    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 01:30:03.064934 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.064911    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"
Sep 11 01:30:03.089609 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.089586    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"
Sep 11 01:30:03.903274 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.903243    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"
Sep 11 01:30:03.903726 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.903242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5" event={"ID":"43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18","Type":"ContainerDied","Data":"65ebbacfb25b59f5928c6ef3d3247f925bdea13e4976d444320d4962ff159253"}
Sep 11 01:30:03.903726 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.903367    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ebbacfb25b59f5928c6ef3d3247f925bdea13e4976d444320d4962ff159253"
Sep 11 01:30:03.904643 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.904616    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"
Sep 11 01:30:03.904769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.904644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv" event={"ID":"d3419d95-37f7-4d46-a5fd-41327d44acc4","Type":"ContainerDied","Data":"264b266245d63b18fed16845328f2513f58d606c29ef2ba98ac2c0a58e8dac0b"}
Sep 11 01:30:03.904769 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.904683    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="264b266245d63b18fed16845328f2513f58d606c29ef2ba98ac2c0a58e8dac0b"
Sep 11 01:30:03.906976 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.906953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" event={"ID":"421d2f92-ddcf-4b84-ba3e-fc605947f26c","Type":"ContainerStarted","Data":"b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14"}
Sep 11 01:30:03.915962 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.915941    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:30:03.916330 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.916297    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 11 01:30:03.916381 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:03.916354    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 11 01:30:04.916219 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:04.916188    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 11 01:30:04.916605 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:04.916241    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 11 01:30:05.916165 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:05.916129    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 11 01:30:05.916396 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:05.916188    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 11 01:30:06.916959 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:06.916922    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 11 01:30:06.917378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:06.916979    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 11 01:30:07.916619 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:07.916577    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 11 01:30:07.916896 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:07.916655    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 11 01:30:08.916509 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:08.916465    2583 patch_prober.go:28] interesting pod/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 11 01:30:08.917026 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:08.916520    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 11 01:30:09.919777 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:09.919744    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:30:09.920186 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:09.920119    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:30:09.921243 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:09.921222    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:30:24.477437 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.477403    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"]
Sep 11 01:30:24.477904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.477830    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18" containerName="cleanup"
Sep 11 01:30:24.477904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.477844    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18" containerName="cleanup"
Sep 11 01:30:24.477904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.477860    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3419d95-37f7-4d46-a5fd-41327d44acc4" containerName="cleanup"
Sep 11 01:30:24.477904 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.477866    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3419d95-37f7-4d46-a5fd-41327d44acc4" containerName="cleanup"
Sep 11 01:30:24.478038 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.477928    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3419d95-37f7-4d46-a5fd-41327d44acc4" containerName="cleanup"
Sep 11 01:30:24.478038 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.477937    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18" containerName="cleanup"
Sep 11 01:30:24.480290 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.480274    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.482612 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.482588    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c1edc6\"/\"openshift-service-ca.crt\""
Sep 11 01:30:24.482612 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.482617    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c1edc6\"/\"kube-root-ca.crt\""
Sep 11 01:30:24.483751 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.483732    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c1edc6\"/\"ipp-route-c1edc6-kserve-self-signed-certs\""
Sep 11 01:30:24.483867 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.483750    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c1edc6\"/\"default-dockercfg-jh5z7\""
Sep 11 01:30:24.491240 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.491220    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"]
Sep 11 01:30:24.618495 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.618457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-dshm\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.618495 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.618499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tmp-dir\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.618715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.618521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tls-certs\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.618715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.618626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-model-cache\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.618715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.618654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kds\" (UniqueName: \"kubernetes.io/projected/469e8db3-d8bb-4e98-9127-6c7f70c4926b-kube-api-access-k5kds\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.618715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.618685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-home\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.719298 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.719265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kds\" (UniqueName: \"kubernetes.io/projected/469e8db3-d8bb-4e98-9127-6c7f70c4926b-kube-api-access-k5kds\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.719661 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.719636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-home\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.719761 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.719744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-dshm\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.719827 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.719787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tmp-dir\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.719882 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.719831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tls-certs\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.719938 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.719928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-model-cache\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.720093 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.720044    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-home\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.720290 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.720238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-model-cache\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.720388 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.720338    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tmp-dir\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.722096 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.722074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-dshm\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.722884 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.722841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tls-certs\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.727385 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.727366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kds\" (UniqueName: \"kubernetes.io/projected/469e8db3-d8bb-4e98-9127-6c7f70c4926b-kube-api-access-k5kds\") pod \"ipp-route-c1edc6-kserve-7945bd89f5-v5gr4\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") " pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:24.791574 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:24.791514    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:25.062844 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:25.062814    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"]
Sep 11 01:30:25.064011 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:30:25.063978    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod469e8db3_d8bb_4e98_9127_6c7f70c4926b.slice/crio-c0b8171c048810b1209bca3ed533ebc8274f5449aa3d1e02d9d8dac656d8b2f0 WatchSource:0}: Error finding container c0b8171c048810b1209bca3ed533ebc8274f5449aa3d1e02d9d8dac656d8b2f0: Status 404 returned error can't find the container with id c0b8171c048810b1209bca3ed533ebc8274f5449aa3d1e02d9d8dac656d8b2f0
Sep 11 01:30:26.031083 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:26.031040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4" event={"ID":"469e8db3-d8bb-4e98-9127-6c7f70c4926b","Type":"ContainerStarted","Data":"2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0"}
Sep 11 01:30:26.031584 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:26.031095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4" event={"ID":"469e8db3-d8bb-4e98-9127-6c7f70c4926b","Type":"ContainerStarted","Data":"c0b8171c048810b1209bca3ed533ebc8274f5449aa3d1e02d9d8dac656d8b2f0"}
Sep 11 01:30:26.047797 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:26.047755    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4" podStartSLOduration=2.047741404 podStartE2EDuration="2.047741404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:30:26.047503184 +0000 UTC m=+1955.584154376" watchObservedRunningTime="2026-09-11 01:30:26.047741404 +0000 UTC m=+1955.584392604"
Sep 11 01:30:28.390829 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:28.390792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:30:34.792329 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:34.792273    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:34.792804 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:34.792352    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:34.804746 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:34.804719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:35.086104 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:35.086073    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:30:41.186359 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:41.186322    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:30:53.819822 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.819733    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"]
Sep 11 01:30:53.824619 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.824595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:53.827496 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.827470    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8fa57c-dockercfg-gwlws\""
Sep 11 01:30:53.827724 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.827708    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8fa57c\""
Sep 11 01:30:53.834913 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.834893    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"]
Sep 11 01:30:53.897185 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.897154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5826a-abbf-4350-85fc-e39a87f02b2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq\" (UID: \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:53.897378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.897213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbjp\" (UniqueName: \"kubernetes.io/projected/d7c5826a-abbf-4350-85fc-e39a87f02b2d-kube-api-access-nzbjp\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq\" (UID: \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:53.998151 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.998114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbjp\" (UniqueName: \"kubernetes.io/projected/d7c5826a-abbf-4350-85fc-e39a87f02b2d-kube-api-access-nzbjp\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq\" (UID: \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:53.998342 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.998202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5826a-abbf-4350-85fc-e39a87f02b2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq\" (UID: \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:53.998842 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:53.998824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5826a-abbf-4350-85fc-e39a87f02b2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq\" (UID: \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:54.007047 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:54.007018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbjp\" (UniqueName: \"kubernetes.io/projected/d7c5826a-abbf-4350-85fc-e39a87f02b2d-kube-api-access-nzbjp\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq\" (UID: \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:54.136445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:54.136361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:54.407677 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:54.407647    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"]
Sep 11 01:30:54.409481 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:30:54.409454    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c5826a_abbf_4350_85fc_e39a87f02b2d.slice/crio-3f214ef39dd9e755099ba7b8ec08cf72e283c12808e5cde2a19d2438d3876a0f WatchSource:0}: Error finding container 3f214ef39dd9e755099ba7b8ec08cf72e283c12808e5cde2a19d2438d3876a0f: Status 404 returned error can't find the container with id 3f214ef39dd9e755099ba7b8ec08cf72e283c12808e5cde2a19d2438d3876a0f
Sep 11 01:30:55.175668 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:55.175630    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq" event={"ID":"d7c5826a-abbf-4350-85fc-e39a87f02b2d","Type":"ContainerStarted","Data":"2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5"}
Sep 11 01:30:55.176067 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:55.175681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq" event={"ID":"d7c5826a-abbf-4350-85fc-e39a87f02b2d","Type":"ContainerStarted","Data":"3f214ef39dd9e755099ba7b8ec08cf72e283c12808e5cde2a19d2438d3876a0f"}
Sep 11 01:30:55.176067 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:55.175702    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:30:55.229377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:30:55.229328    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq" podStartSLOduration=2.22929753 podStartE2EDuration="2.22929753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:30:55.225753141 +0000 UTC m=+1984.762404342" watchObservedRunningTime="2026-09-11 01:30:55.22929753 +0000 UTC m=+1984.765948731"
Sep 11 01:31:01.586915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:01.586873    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:31:06.182499 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:06.182471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:31:23.856776 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:23.856721    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"]
Sep 11 01:31:23.857250 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:23.857013    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq" podUID="d7c5826a-abbf-4350-85fc-e39a87f02b2d" containerName="payload-pre-processing" containerID="cri-o://2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5" gracePeriod=30
Sep 11 01:31:24.109823 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.109759    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:31:24.182727 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.182696    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzbjp\" (UniqueName: \"kubernetes.io/projected/d7c5826a-abbf-4350-85fc-e39a87f02b2d-kube-api-access-nzbjp\") pod \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\" (UID: \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\") "
Sep 11 01:31:24.182890 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.182735    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5826a-abbf-4350-85fc-e39a87f02b2d-plugins-config-volume\") pod \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\" (UID: \"d7c5826a-abbf-4350-85fc-e39a87f02b2d\") "
Sep 11 01:31:24.183130 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.183107    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c5826a-abbf-4350-85fc-e39a87f02b2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7c5826a-abbf-4350-85fc-e39a87f02b2d" (UID: "d7c5826a-abbf-4350-85fc-e39a87f02b2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:24.184836 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.184810    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c5826a-abbf-4350-85fc-e39a87f02b2d-kube-api-access-nzbjp" (OuterVolumeSpecName: "kube-api-access-nzbjp") pod "d7c5826a-abbf-4350-85fc-e39a87f02b2d" (UID: "d7c5826a-abbf-4350-85fc-e39a87f02b2d"). InnerVolumeSpecName "kube-api-access-nzbjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:31:24.284263 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.284223    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbjp\" (UniqueName: \"kubernetes.io/projected/d7c5826a-abbf-4350-85fc-e39a87f02b2d-kube-api-access-nzbjp\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:31:24.284263 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.284260    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5826a-abbf-4350-85fc-e39a87f02b2d-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:31:24.318076 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.318046    2583 generic.go:356] "Generic (PLEG): container finished" podID="d7c5826a-abbf-4350-85fc-e39a87f02b2d" containerID="2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5" exitCode=0
Sep 11 01:31:24.318191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.318091    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"
Sep 11 01:31:24.318191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.318134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq" event={"ID":"d7c5826a-abbf-4350-85fc-e39a87f02b2d","Type":"ContainerDied","Data":"2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5"}
Sep 11 01:31:24.318191 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.318182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq" event={"ID":"d7c5826a-abbf-4350-85fc-e39a87f02b2d","Type":"ContainerDied","Data":"3f214ef39dd9e755099ba7b8ec08cf72e283c12808e5cde2a19d2438d3876a0f"}
Sep 11 01:31:24.318298 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.318210    2583 scope.go:117] "RemoveContainer" containerID="2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5"
Sep 11 01:31:24.327239 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.327224    2583 scope.go:117] "RemoveContainer" containerID="2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5"
Sep 11 01:31:24.327521 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:31:24.327503    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5\": container with ID starting with 2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5 not found: ID does not exist" containerID="2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5"
Sep 11 01:31:24.327598 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.327530    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5"} err="failed to get container status \"2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5\": rpc error: code = NotFound desc = could not find container \"2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5\": container with ID starting with 2dc81a4030a9a7a2c0116c84ec59f61972c5c18d20ad45875eaa1ad6d7b064e5 not found: ID does not exist"
Sep 11 01:31:24.342540 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.342516    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"]
Sep 11 01:31:24.344469 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:24.344447    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fa57c-54dc9bcbb-cjklq"]
Sep 11 01:31:25.037119 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:25.037087    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7c5826a-abbf-4350-85fc-e39a87f02b2d" path="/var/lib/kubelet/pods/d7c5826a-abbf-4350-85fc-e39a87f02b2d/volumes"
Sep 11 01:31:28.286655 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:28.286622    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:31:47.681163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:47.681129    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:31:50.005002 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.004965    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"]
Sep 11 01:31:50.005471 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.005214    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x" podUID="0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" containerName="maas-api" containerID="cri-o://c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271" gracePeriod=30
Sep 11 01:31:50.031470 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.031301    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"]
Sep 11 01:31:50.033069 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.033036    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c1edc6-29818170-jjhmv"]
Sep 11 01:31:50.283294 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.283264    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:31:50.421203 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.421165    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk8s5\" (UniqueName: \"kubernetes.io/projected/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-kube-api-access-tk8s5\") pod \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") "
Sep 11 01:31:50.421378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.421211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-openshift-service-ca\") pod \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") "
Sep 11 01:31:50.421378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.421284    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-metrics-certs\") pod \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") "
Sep 11 01:31:50.421568 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.421380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls\") pod \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\" (UID: \"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0\") "
Sep 11 01:31:50.421688 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.421648    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" (UID: "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:31:50.423952 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.423926    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" (UID: "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:31:50.424048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.423974    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-kube-api-access-tk8s5" (OuterVolumeSpecName: "kube-api-access-tk8s5") pod "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" (UID: "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0"). InnerVolumeSpecName "kube-api-access-tk8s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:31:50.424048 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.424035    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" (UID: "0cd14cd2-b3a5-4169-ba5e-81bcd93912a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:31:50.453602 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.453573    2583 generic.go:356] "Generic (PLEG): container finished" podID="0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" containerID="c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271" exitCode=0
Sep 11 01:31:50.453708 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.453623    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"
Sep 11 01:31:50.453708 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.453636    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x" event={"ID":"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0","Type":"ContainerDied","Data":"c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271"}
Sep 11 01:31:50.453708 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.453676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x" event={"ID":"0cd14cd2-b3a5-4169-ba5e-81bcd93912a0","Type":"ContainerDied","Data":"c41c2f253f3f87d0e00c987a6cedf7d622d83b11994ee87afcaf24ba26ecb0f8"}
Sep 11 01:31:50.453708 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.453696    2583 scope.go:117] "RemoveContainer" containerID="c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271"
Sep 11 01:31:50.463214 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.463198    2583 scope.go:117] "RemoveContainer" containerID="c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271"
Sep 11 01:31:50.463546 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:31:50.463524    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271\": container with ID starting with c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271 not found: ID does not exist" containerID="c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271"
Sep 11 01:31:50.463638 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.463557    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271"} err="failed to get container status \"c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271\": rpc error: code = NotFound desc = could not find container \"c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271\": container with ID starting with c34f5fcc16f577dfd0a5a6497ec700dcea232f56f87a5c24c903b8ae4649a271 not found: ID does not exist"
Sep 11 01:31:50.476004 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.475984    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"]
Sep 11 01:31:50.480199 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.480178    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1edc6-7f74c5f489-54p6x"]
Sep 11 01:31:50.522362 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.522264    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk8s5\" (UniqueName: \"kubernetes.io/projected/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-kube-api-access-tk8s5\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:31:50.522362 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.522293    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-openshift-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:31:50.522362 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.522338    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-metrics-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:31:50.522362 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:50.522352    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0-maas-api-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:31:51.037399 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:51.037364    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" path="/var/lib/kubelet/pods/0cd14cd2-b3a5-4169-ba5e-81bcd93912a0/volumes"
Sep 11 01:31:51.037863 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:51.037788    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3419d95-37f7-4d46-a5fd-41327d44acc4" path="/var/lib/kubelet/pods/d3419d95-37f7-4d46-a5fd-41327d44acc4/volumes"
Sep 11 01:31:55.602581 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:55.602550    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"]
Sep 11 01:31:59.683710 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:31:59.683677    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:32:10.622129 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.622085    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4" podUID="469e8db3-d8bb-4e98-9127-6c7f70c4926b" containerName="main" containerID="cri-o://2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0" gracePeriod=45
Sep 11 01:32:10.873424 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.873367    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:32:10.895469 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.895443    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tls-certs\") pod \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") "
Sep 11 01:32:10.895654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.895629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tmp-dir\") pod \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") "
Sep 11 01:32:10.895753 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.895706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-model-cache\") pod \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") "
Sep 11 01:32:10.895831 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.895760    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-dshm\") pod \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") "
Sep 11 01:32:10.895891 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.895849    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5kds\" (UniqueName: \"kubernetes.io/projected/469e8db3-d8bb-4e98-9127-6c7f70c4926b-kube-api-access-k5kds\") pod \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") "
Sep 11 01:32:10.895891 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.895884    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-home\") pod \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\" (UID: \"469e8db3-d8bb-4e98-9127-6c7f70c4926b\") "
Sep 11 01:32:10.896216 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.896028    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "469e8db3-d8bb-4e98-9127-6c7f70c4926b" (UID: "469e8db3-d8bb-4e98-9127-6c7f70c4926b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:10.896349 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.896324    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-model-cache" (OuterVolumeSpecName: "model-cache") pod "469e8db3-d8bb-4e98-9127-6c7f70c4926b" (UID: "469e8db3-d8bb-4e98-9127-6c7f70c4926b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:10.896564 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.896543    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-home" (OuterVolumeSpecName: "home") pod "469e8db3-d8bb-4e98-9127-6c7f70c4926b" (UID: "469e8db3-d8bb-4e98-9127-6c7f70c4926b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:10.898595 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.898560    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-dshm" (OuterVolumeSpecName: "dshm") pod "469e8db3-d8bb-4e98-9127-6c7f70c4926b" (UID: "469e8db3-d8bb-4e98-9127-6c7f70c4926b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:10.898806 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.898780    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469e8db3-d8bb-4e98-9127-6c7f70c4926b-kube-api-access-k5kds" (OuterVolumeSpecName: "kube-api-access-k5kds") pod "469e8db3-d8bb-4e98-9127-6c7f70c4926b" (UID: "469e8db3-d8bb-4e98-9127-6c7f70c4926b"). InnerVolumeSpecName "kube-api-access-k5kds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:32:10.898905 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.898813    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "469e8db3-d8bb-4e98-9127-6c7f70c4926b" (UID: "469e8db3-d8bb-4e98-9127-6c7f70c4926b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:32:10.997082 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.997052    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-model-cache\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:10.997082 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.997076    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-dshm\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:10.997082 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.997086    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5kds\" (UniqueName: \"kubernetes.io/projected/469e8db3-d8bb-4e98-9127-6c7f70c4926b-kube-api-access-k5kds\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:10.997377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.997100    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-home\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:10.997377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.997112    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tls-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:10.997377 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:10.997126    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469e8db3-d8bb-4e98-9127-6c7f70c4926b-tmp-dir\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:11.557911 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.557874    2583 generic.go:356] "Generic (PLEG): container finished" podID="469e8db3-d8bb-4e98-9127-6c7f70c4926b" containerID="2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0" exitCode=0
Sep 11 01:32:11.558112 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.557935    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"
Sep 11 01:32:11.558112 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.557955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4" event={"ID":"469e8db3-d8bb-4e98-9127-6c7f70c4926b","Type":"ContainerDied","Data":"2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0"}
Sep 11 01:32:11.558112 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.557993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4" event={"ID":"469e8db3-d8bb-4e98-9127-6c7f70c4926b","Type":"ContainerDied","Data":"c0b8171c048810b1209bca3ed533ebc8274f5449aa3d1e02d9d8dac656d8b2f0"}
Sep 11 01:32:11.558112 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.558009    2583 scope.go:117] "RemoveContainer" containerID="2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0"
Sep 11 01:32:11.566376 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.566360    2583 scope.go:117] "RemoveContainer" containerID="2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0"
Sep 11 01:32:11.566648 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:32:11.566629    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0\": container with ID starting with 2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0 not found: ID does not exist" containerID="2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0"
Sep 11 01:32:11.566717 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.566658    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0"} err="failed to get container status \"2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0\": rpc error: code = NotFound desc = could not find container \"2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0\": container with ID starting with 2b3eb8c9d69dbc86dd6956545146609867d7967ad4f50cb5005109aa71cb7fe0 not found: ID does not exist"
Sep 11 01:32:11.573460 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.573437    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"]
Sep 11 01:32:11.577053 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:11.577032    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c1edc6/ipp-route-c1edc6-kserve-7945bd89f5-v5gr4"]
Sep 11 01:32:13.036915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:13.036884    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="469e8db3-d8bb-4e98-9127-6c7f70c4926b" path="/var/lib/kubelet/pods/469e8db3-d8bb-4e98-9127-6c7f70c4926b/volumes"
Sep 11 01:32:38.885599 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:38.885565    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"]
Sep 11 01:32:38.886150 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:38.885774    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy" containerID="cri-o://76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630" gracePeriod=30
Sep 11 01:32:39.237389 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.237287    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 01:32:39.237557 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.237378    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 01:32:39.293338 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293288    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"]
Sep 11 01:32:39.293663 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293650    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469e8db3-d8bb-4e98-9127-6c7f70c4926b" containerName="main"
Sep 11 01:32:39.293712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293664    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="469e8db3-d8bb-4e98-9127-6c7f70c4926b" containerName="main"
Sep 11 01:32:39.293712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293679    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" containerName="maas-api"
Sep 11 01:32:39.293712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293684    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" containerName="maas-api"
Sep 11 01:32:39.293712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293695    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c5826a-abbf-4350-85fc-e39a87f02b2d" containerName="payload-pre-processing"
Sep 11 01:32:39.293712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293703    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c5826a-abbf-4350-85fc-e39a87f02b2d" containerName="payload-pre-processing"
Sep 11 01:32:39.293890 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293759    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c5826a-abbf-4350-85fc-e39a87f02b2d" containerName="payload-pre-processing"
Sep 11 01:32:39.293890 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293768    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="469e8db3-d8bb-4e98-9127-6c7f70c4926b" containerName="main"
Sep 11 01:32:39.293890 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.293775    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd14cd2-b3a5-4169-ba5e-81bcd93912a0" containerName="maas-api"
Sep 11 01:32:39.296788 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.296769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"
Sep 11 01:32:39.298937 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.298918    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q6gbn\""
Sep 11 01:32:39.299176 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.299160    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:32:39.303579 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.303554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"]
Sep 11 01:32:39.442362 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.442298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd579832-3314-482c-aea5-d9460356fe8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88\" (UID: \"cd579832-3314-482c-aea5-d9460356fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"
Sep 11 01:32:39.543597 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.543515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd579832-3314-482c-aea5-d9460356fe8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88\" (UID: \"cd579832-3314-482c-aea5-d9460356fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"
Sep 11 01:32:39.544121 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.544103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd579832-3314-482c-aea5-d9460356fe8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88\" (UID: \"cd579832-3314-482c-aea5-d9460356fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"
Sep 11 01:32:39.607677 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.607652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"
Sep 11 01:32:39.880086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:39.880059    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"]
Sep 11 01:32:39.882185 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:32:39.882160    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd579832_3314_482c_aea5_d9460356fe8d.slice/crio-7ca2697d2c9b932fa14b7909e7fe516d686346ea79c970846832678cf85a24c6 WatchSource:0}: Error finding container 7ca2697d2c9b932fa14b7909e7fe516d686346ea79c970846832678cf85a24c6: Status 404 returned error can't find the container with id 7ca2697d2c9b932fa14b7909e7fe516d686346ea79c970846832678cf85a24c6
Sep 11 01:32:40.700915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:40.700874    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd579832-3314-482c-aea5-d9460356fe8d" containerID="dc648402e7be9591acb2c130f70b839ed7d7c830d6fbbef0a472a108844dd631" exitCode=0
Sep 11 01:32:40.701362 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:40.700941    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88" event={"ID":"cd579832-3314-482c-aea5-d9460356fe8d","Type":"ContainerDied","Data":"dc648402e7be9591acb2c130f70b839ed7d7c830d6fbbef0a472a108844dd631"}
Sep 11 01:32:40.701362 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:40.700984    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88" event={"ID":"cd579832-3314-482c-aea5-d9460356fe8d","Type":"ContainerStarted","Data":"7ca2697d2c9b932fa14b7909e7fe516d686346ea79c970846832678cf85a24c6"}
Sep 11 01:32:41.835203 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:41.835177    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"
Sep 11 01:32:41.962622 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:41.962549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd579832-3314-482c-aea5-d9460356fe8d-maas-api-service-ca\") pod \"cd579832-3314-482c-aea5-d9460356fe8d\" (UID: \"cd579832-3314-482c-aea5-d9460356fe8d\") "
Sep 11 01:32:41.962867 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:41.962843    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd579832-3314-482c-aea5-d9460356fe8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd579832-3314-482c-aea5-d9460356fe8d" (UID: "cd579832-3314-482c-aea5-d9460356fe8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:32:42.063126 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:42.063095    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd579832-3314-482c-aea5-d9460356fe8d-maas-api-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:42.711781 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:42.711753    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"
Sep 11 01:32:42.711781 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:42.711766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88" event={"ID":"cd579832-3314-482c-aea5-d9460356fe8d","Type":"ContainerDied","Data":"7ca2697d2c9b932fa14b7909e7fe516d686346ea79c970846832678cf85a24c6"}
Sep 11 01:32:42.711988 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:42.711808    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca2697d2c9b932fa14b7909e7fe516d686346ea79c970846832678cf85a24c6"
Sep 11 01:32:44.135370 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.135346    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:32:44.283882 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.283799    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8njq7\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-kube-api-access-8njq7\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.283882 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.283847    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-certs\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.283882 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.283876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-data\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.284142 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.283902    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-podinfo\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.284142 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.283933    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-envoy\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.284142 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.283991    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-token\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.284142 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284027    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-credential-socket\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.284142 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istiod-ca-cert\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.284142 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-socket\") pod \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\" (UID: \"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e\") "
Sep 11 01:32:44.284503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:44.284503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284161    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:44.284503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284336    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:44.284503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284430    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.284503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284448    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.284503 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284463    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.284829 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284570    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:44.284829 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.284694    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:32:44.286265 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.286235    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:32:44.286265 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.286234    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:32:44.286476 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.286453    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:32:44.286561 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.286541    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-kube-api-access-8njq7" (OuterVolumeSpecName: "kube-api-access-8njq7") pod "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" (UID: "7ef7c2c7-9857-4238-94a5-bb1ed0d5420e"). InnerVolumeSpecName "kube-api-access-8njq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:32:44.385071 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.385036    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.385071 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.385063    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.385071 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.385072    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.385384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.385080    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8njq7\" (UniqueName: \"kubernetes.io/projected/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-kube-api-access-8njq7\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.385384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.385091    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.385384 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.385099    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:32:44.723692 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.723660    2583 generic.go:356] "Generic (PLEG): container finished" podID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerID="76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630" exitCode=0
Sep 11 01:32:44.723875 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.723743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" event={"ID":"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e","Type":"ContainerDied","Data":"76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630"}
Sep 11 01:32:44.724561 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.724055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w" event={"ID":"7ef7c2c7-9857-4238-94a5-bb1ed0d5420e","Type":"ContainerDied","Data":"4ac2c255942723f9e64ddd55d41041d15b58ec72320c0b84a96486099e480cd2"}
Sep 11 01:32:44.724561 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.724184    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"
Sep 11 01:32:44.724561 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.724371    2583 scope.go:117] "RemoveContainer" containerID="76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630"
Sep 11 01:32:44.736577 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.736560    2583 scope.go:117] "RemoveContainer" containerID="3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1"
Sep 11 01:32:44.746492 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.746472    2583 scope.go:117] "RemoveContainer" containerID="76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630"
Sep 11 01:32:44.746964 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:32:44.746938    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630\": container with ID starting with 76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630 not found: ID does not exist" containerID="76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630"
Sep 11 01:32:44.747073 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.746973    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630"} err="failed to get container status \"76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630\": rpc error: code = NotFound desc = could not find container \"76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630\": container with ID starting with 76180e3ff676158a06d67ba6db123ffb9d20ec39514c6321e717d58b7e9da630 not found: ID does not exist"
Sep 11 01:32:44.747073 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.746994    2583 scope.go:117] "RemoveContainer" containerID="3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1"
Sep 11 01:32:44.747270 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:32:44.747255    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1\": container with ID starting with 3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1 not found: ID does not exist" containerID="3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1"
Sep 11 01:32:44.747404 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.747275    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1"} err="failed to get container status \"3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1\": rpc error: code = NotFound desc = could not find container \"3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1\": container with ID starting with 3647354d7ddc64f9239fe56eeea21df72b4f335ce52000f6399081a9a6a53fe1 not found: ID does not exist"
Sep 11 01:32:44.748830 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.748811    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"]
Sep 11 01:32:44.752078 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:44.752051    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c1edc6-openshift-default-6c98cd4f94-lq27w"]
Sep 11 01:32:45.037096 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:45.037004    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" path="/var/lib/kubelet/pods/7ef7c2c7-9857-4238-94a5-bb1ed0d5420e/volumes"
Sep 11 01:32:50.094797 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:50.094758    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"]
Sep 11 01:32:50.097228 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:50.097200    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40f90a-7hb88"]
Sep 11 01:32:51.037662 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:51.037630    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd579832-3314-482c-aea5-d9460356fe8d" path="/var/lib/kubelet/pods/cd579832-3314-482c-aea5-d9460356fe8d/volumes"
Sep 11 01:32:51.038562 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:51.038541    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:32:51.041211 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:51.041188    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:32:51.044078 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:51.044058    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:32:51.046484 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:51.046464    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:32:53.584579 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:53.584543    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:32:56.482247 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:56.482211    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"]
Sep 11 01:32:56.482659 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:32:56.482487    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy" containerID="cri-o://b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14" gracePeriod=30
Sep 11 01:33:01.748585 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.748522    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:33:01.804281 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.804242    2583 generic.go:356] "Generic (PLEG): container finished" podID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerID="b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14" exitCode=0
Sep 11 01:33:01.804493 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.804295    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"
Sep 11 01:33:01.804493 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.804369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" event={"ID":"421d2f92-ddcf-4b84-ba3e-fc605947f26c","Type":"ContainerDied","Data":"b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14"}
Sep 11 01:33:01.804493 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.804483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj" event={"ID":"421d2f92-ddcf-4b84-ba3e-fc605947f26c","Type":"ContainerDied","Data":"4eab6fccb4df3479e175f27f1c25f92e6d2a22aa8fc8a60fcf1ed930138a43c3"}
Sep 11 01:33:01.804671 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.804519    2583 scope.go:117] "RemoveContainer" containerID="b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14"
Sep 11 01:33:01.813052 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.813037    2583 scope.go:117] "RemoveContainer" containerID="817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780"
Sep 11 01:33:01.820650 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.820626    2583 scope.go:117] "RemoveContainer" containerID="b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14"
Sep 11 01:33:01.820919 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:33:01.820901    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14\": container with ID starting with b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14 not found: ID does not exist" containerID="b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14"
Sep 11 01:33:01.820974 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.820928    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14"} err="failed to get container status \"b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14\": rpc error: code = NotFound desc = could not find container \"b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14\": container with ID starting with b3d53ed97e107881b79ce7e884e81beb52395ebb4a7a4cbb5d084aec9bfdda14 not found: ID does not exist"
Sep 11 01:33:01.820974 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.820943    2583 scope.go:117] "RemoveContainer" containerID="817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780"
Sep 11 01:33:01.821188 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:33:01.821174    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780\": container with ID starting with 817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780 not found: ID does not exist" containerID="817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780"
Sep 11 01:33:01.821232 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.821191    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780"} err="failed to get container status \"817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780\": rpc error: code = NotFound desc = could not find container \"817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780\": container with ID starting with 817e2a4cc1d98dbce1e53c5a035cf93790744a72439d7e84b91c9ef9bf18d780 not found: ID does not exist"
Sep 11 01:33:01.835598 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-token\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.835598 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835606    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-socket\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.835767 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835633    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv2cc\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-kube-api-access-cv2cc\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.835767 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835656    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-podinfo\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.835767 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835728    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-credential-socket\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.835767 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istiod-ca-cert\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.836034 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-certs\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.836034 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835832    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:33:01.836034 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835853    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-envoy\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.836034 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.835882    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-data\") pod \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\" (UID: \"421d2f92-ddcf-4b84-ba3e-fc605947f26c\") "
Sep 11 01:33:01.836034 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.836020    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:33:01.836034 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.836032    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:33:01.836418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.836138    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:33:01.836418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.836241    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-data" (OuterVolumeSpecName: "istio-data") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:33:01.836418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.836339    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istiod-ca-cert\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:01.836418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.836361    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:01.836418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.836374    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-workload-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:01.836418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.836387    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-credential-socket\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:01.837745 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.837726    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-token" (OuterVolumeSpecName: "istio-token") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:33:01.837838 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.837808    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-kube-api-access-cv2cc" (OuterVolumeSpecName: "kube-api-access-cv2cc") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "kube-api-access-cv2cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:33:01.837893 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.837875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 01:33:01.837977 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.837956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "421d2f92-ddcf-4b84-ba3e-fc605947f26c" (UID: "421d2f92-ddcf-4b84-ba3e-fc605947f26c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:33:01.936823 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.936784    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-token\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:01.936823 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.936821    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2cc\" (UniqueName: \"kubernetes.io/projected/421d2f92-ddcf-4b84-ba3e-fc605947f26c-kube-api-access-cv2cc\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:01.936823 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.936831    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-podinfo\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:01.936823 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.936840    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-envoy\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:01.937064 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:01.936849    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421d2f92-ddcf-4b84-ba3e-fc605947f26c-istio-data\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:02.141880 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:02.141850    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"]
Sep 11 01:33:02.144746 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:02.144726    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-40f90a-openshift-default-c6dc47dc6-kr8vj"]
Sep 11 01:33:03.036876 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:03.036844    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" path="/var/lib/kubelet/pods/421d2f92-ddcf-4b84-ba3e-fc605947f26c/volumes"
Sep 11 01:33:07.418639 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:07.418552    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"]
Sep 11 01:33:07.424366 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:07.424331    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d6e26b-29818170-6tvp5"]
Sep 11 01:33:09.037461 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:09.037424    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18" path="/var/lib/kubelet/pods/43c7696b-a8a7-4fdc-9d6f-1faa6f11fe18/volumes"
Sep 11 01:33:10.481220 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:10.481184    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:33:25.180205 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:25.180164    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"]
Sep 11 01:33:25.922225 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:25.922187    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7" podUID="14698a22-6a8c-471c-a67e-8045c43816d5" containerName="payload-processing" containerID="cri-o://16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a" gracePeriod=30
Sep 11 01:33:26.171996 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.171970    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:33:26.238955 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.238875    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c94p\" (UniqueName: \"kubernetes.io/projected/14698a22-6a8c-471c-a67e-8045c43816d5-kube-api-access-7c94p\") pod \"14698a22-6a8c-471c-a67e-8045c43816d5\" (UID: \"14698a22-6a8c-471c-a67e-8045c43816d5\") "
Sep 11 01:33:26.238955 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.238928    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14698a22-6a8c-471c-a67e-8045c43816d5-plugins-config-volume\") pod \"14698a22-6a8c-471c-a67e-8045c43816d5\" (UID: \"14698a22-6a8c-471c-a67e-8045c43816d5\") "
Sep 11 01:33:26.239414 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.239382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14698a22-6a8c-471c-a67e-8045c43816d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14698a22-6a8c-471c-a67e-8045c43816d5" (UID: "14698a22-6a8c-471c-a67e-8045c43816d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:33:26.241352 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.241320    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14698a22-6a8c-471c-a67e-8045c43816d5-kube-api-access-7c94p" (OuterVolumeSpecName: "kube-api-access-7c94p") pod "14698a22-6a8c-471c-a67e-8045c43816d5" (UID: "14698a22-6a8c-471c-a67e-8045c43816d5"). InnerVolumeSpecName "kube-api-access-7c94p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:33:26.340420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.340386    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c94p\" (UniqueName: \"kubernetes.io/projected/14698a22-6a8c-471c-a67e-8045c43816d5-kube-api-access-7c94p\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:26.340420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.340412    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14698a22-6a8c-471c-a67e-8045c43816d5-plugins-config-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:33:26.927931 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.927899    2583 generic.go:356] "Generic (PLEG): container finished" podID="14698a22-6a8c-471c-a67e-8045c43816d5" containerID="16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a" exitCode=0
Sep 11 01:33:26.928116 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.927944    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"
Sep 11 01:33:26.928116 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.927963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7" event={"ID":"14698a22-6a8c-471c-a67e-8045c43816d5","Type":"ContainerDied","Data":"16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a"}
Sep 11 01:33:26.928116 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.928004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7" event={"ID":"14698a22-6a8c-471c-a67e-8045c43816d5","Type":"ContainerDied","Data":"15bc36e86d9403b9eedb65c5817ca05c21f9c494aa92bcb09f04737565bb6241"}
Sep 11 01:33:26.928116 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.928026    2583 scope.go:117] "RemoveContainer" containerID="16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a"
Sep 11 01:33:26.937635 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.937613    2583 scope.go:117] "RemoveContainer" containerID="16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a"
Sep 11 01:33:26.937895 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:33:26.937879    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a\": container with ID starting with 16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a not found: ID does not exist" containerID="16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a"
Sep 11 01:33:26.937963 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.937903    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a"} err="failed to get container status \"16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a\": rpc error: code = NotFound desc = could not find container \"16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a\": container with ID starting with 16a46889f14d77cf4c6eb207964821f3e14434bfc3e0b99ecc3c2d3d2d06b89a not found: ID does not exist"
Sep 11 01:33:26.948670 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.948642    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"]
Sep 11 01:33:26.952057 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:26.952036    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b9930f-6cbdc6f967-879w7"]
Sep 11 01:33:27.037884 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:27.037853    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14698a22-6a8c-471c-a67e-8045c43816d5" path="/var/lib/kubelet/pods/14698a22-6a8c-471c-a67e-8045c43816d5/volumes"
Sep 11 01:33:27.882894 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:27.882860    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:33:37.173194 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:33:37.173158    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:34:59.681262 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:34:59.681226    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:35:26.483846 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:35:26.483766    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:35:34.377961 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:35:34.377926    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:35:41.477598 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:35:41.477566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:35:49.076101 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:35:49.076065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:36:03.776981 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:03.776948    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:36:11.575707 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:11.575672    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:36:17.079374 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.079303    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"]
Sep 11 01:36:17.079909 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.079609    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn" podUID="0cd59c8e-dff7-4344-9146-f1f2d1411d55" containerName="manager" containerID="cri-o://2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4" gracePeriod=10
Sep 11 01:36:17.539031 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.539004    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:36:17.625985 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.625852    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rw55\" (UniqueName: \"kubernetes.io/projected/0cd59c8e-dff7-4344-9146-f1f2d1411d55-kube-api-access-4rw55\") pod \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\" (UID: \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\") "
Sep 11 01:36:17.625985 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.625972    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cd59c8e-dff7-4344-9146-f1f2d1411d55-extensions-socket-volume\") pod \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\" (UID: \"0cd59c8e-dff7-4344-9146-f1f2d1411d55\") "
Sep 11 01:36:17.626391 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.626348    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd59c8e-dff7-4344-9146-f1f2d1411d55-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0cd59c8e-dff7-4344-9146-f1f2d1411d55" (UID: "0cd59c8e-dff7-4344-9146-f1f2d1411d55"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 01:36:17.627918 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.627889    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd59c8e-dff7-4344-9146-f1f2d1411d55-kube-api-access-4rw55" (OuterVolumeSpecName: "kube-api-access-4rw55") pod "0cd59c8e-dff7-4344-9146-f1f2d1411d55" (UID: "0cd59c8e-dff7-4344-9146-f1f2d1411d55"). InnerVolumeSpecName "kube-api-access-4rw55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:36:17.726931 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.726899    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rw55\" (UniqueName: \"kubernetes.io/projected/0cd59c8e-dff7-4344-9146-f1f2d1411d55-kube-api-access-4rw55\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:36:17.726931 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.726924    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cd59c8e-dff7-4344-9146-f1f2d1411d55-extensions-socket-volume\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:36:17.749146 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.749115    2583 generic.go:356] "Generic (PLEG): container finished" podID="0cd59c8e-dff7-4344-9146-f1f2d1411d55" containerID="2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4" exitCode=0
Sep 11 01:36:17.749270 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.749164    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"
Sep 11 01:36:17.749270 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.749209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn" event={"ID":"0cd59c8e-dff7-4344-9146-f1f2d1411d55","Type":"ContainerDied","Data":"2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4"}
Sep 11 01:36:17.749270 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.749262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn" event={"ID":"0cd59c8e-dff7-4344-9146-f1f2d1411d55","Type":"ContainerDied","Data":"b33bf5b27d205f8b9778a1a092807f3ea9f57599e3921305fff34c31bcc6df08"}
Sep 11 01:36:17.749422 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.749285    2583 scope.go:117] "RemoveContainer" containerID="2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4"
Sep 11 01:36:17.758282 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.758261    2583 scope.go:117] "RemoveContainer" containerID="2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4"
Sep 11 01:36:17.758563 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:36:17.758543    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4\": container with ID starting with 2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4 not found: ID does not exist" containerID="2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4"
Sep 11 01:36:17.758615 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.758572    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4"} err="failed to get container status \"2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4\": rpc error: code = NotFound desc = could not find container \"2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4\": container with ID starting with 2dd67da12c42b51b57a5d49caf5614a95e5c1758a3ef95777ce912c62d5736a4 not found: ID does not exist"
Sep 11 01:36:17.771329 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.771293    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"]
Sep 11 01:36:17.775326 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:17.775292    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bx6xn"]
Sep 11 01:36:19.037317 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:19.037269    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd59c8e-dff7-4344-9146-f1f2d1411d55" path="/var/lib/kubelet/pods/0cd59c8e-dff7-4344-9146-f1f2d1411d55/volumes"
Sep 11 01:36:51.307071 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:51.306993    2583 scope.go:117] "RemoveContainer" containerID="f2f394191ff742b90517be3ffef8cf14ba949dcbeaad6a5cdb625a25f54e2c61"
Sep 11 01:36:51.315124 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:36:51.315103    2583 scope.go:117] "RemoveContainer" containerID="7d79f060f9b8c642788afd5aa1b7db3b3de5d5c3d590275ba45072904117ba9a"
Sep 11 01:37:17.803366 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803334    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"]
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803682    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803694    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803702    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd579832-3314-482c-aea5-d9460356fe8d" containerName="revoke-keys"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803708    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd579832-3314-482c-aea5-d9460356fe8d" containerName="revoke-keys"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803724    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803730    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803745    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd59c8e-dff7-4344-9146-f1f2d1411d55" containerName="manager"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803751    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd59c8e-dff7-4344-9146-f1f2d1411d55" containerName="manager"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803757    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14698a22-6a8c-471c-a67e-8045c43816d5" containerName="payload-processing"
Sep 11 01:37:17.803793 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803763    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="14698a22-6a8c-471c-a67e-8045c43816d5" containerName="payload-processing"
Sep 11 01:37:17.804140 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803816    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd59c8e-dff7-4344-9146-f1f2d1411d55" containerName="manager"
Sep 11 01:37:17.804140 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803828    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy"
Sep 11 01:37:17.804140 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803834    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy"
Sep 11 01:37:17.804140 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803839    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="14698a22-6a8c-471c-a67e-8045c43816d5" containerName="payload-processing"
Sep 11 01:37:17.804140 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803845    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd579832-3314-482c-aea5-d9460356fe8d" containerName="revoke-keys"
Sep 11 01:37:17.804140 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803851    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy"
Sep 11 01:37:17.804140 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.803856    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy"
Sep 11 01:37:17.806797 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.806781    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:17.811073 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.811057    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fb2j7\""
Sep 11 01:37:17.819002 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.818979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"]
Sep 11 01:37:17.930555 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.930523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppsn\" (UniqueName: \"kubernetes.io/projected/d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5-kube-api-access-hppsn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2f8dz\" (UID: \"d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:17.930714 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:17.930612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2f8dz\" (UID: \"d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:18.031201 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:18.031161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppsn\" (UniqueName: \"kubernetes.io/projected/d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5-kube-api-access-hppsn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2f8dz\" (UID: \"d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:18.031403 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:18.031258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2f8dz\" (UID: \"d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:18.031657 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:18.031637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2f8dz\" (UID: \"d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:18.042728 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:18.042705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppsn\" (UniqueName: \"kubernetes.io/projected/d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5-kube-api-access-hppsn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2f8dz\" (UID: \"d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:18.116068 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:18.116029    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:18.392968 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:18.392944    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"]
Sep 11 01:37:18.394969 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:37:18.394935    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd482aff3_78b1_4d5b_aa1d_417c4a9fb8c5.slice/crio-f9081d1e4427af2d9dc451b5131045d776488549c4ae328f59a2091fa12cdeac WatchSource:0}: Error finding container f9081d1e4427af2d9dc451b5131045d776488549c4ae328f59a2091fa12cdeac: Status 404 returned error can't find the container with id f9081d1e4427af2d9dc451b5131045d776488549c4ae328f59a2091fa12cdeac
Sep 11 01:37:19.037539 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:19.037498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz" event={"ID":"d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5","Type":"ContainerStarted","Data":"5151f20231ce61fe53b457157c82fee18eebd01808c35789b9be70776a1ee2f4"}
Sep 11 01:37:19.038029 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:19.037566    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:19.038029 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:19.037594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz" event={"ID":"d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5","Type":"ContainerStarted","Data":"f9081d1e4427af2d9dc451b5131045d776488549c4ae328f59a2091fa12cdeac"}
Sep 11 01:37:19.064208 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:19.060394    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz" podStartSLOduration=2.060366773 podStartE2EDuration="2.060366773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:37:19.056325962 +0000 UTC m=+2368.592977162" watchObservedRunningTime="2026-09-11 01:37:19.060366773 +0000 UTC m=+2368.597017976"
Sep 11 01:37:30.045134 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:30.045100    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2f8dz"
Sep 11 01:37:38.670562 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:38.670524    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:37:45.081532 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:45.081495    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:37:51.066885 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:51.066858    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:37:51.071094 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:51.071072    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:37:51.072568 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:51.072548    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:37:51.076335 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:51.076318    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwgdk_ba79b33e-4621-4d88-8987-dbf54c5f5089/ovn-acl-logging/0.log"
Sep 11 01:37:52.371226 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:37:52.371191    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:38:03.080712 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:03.080674    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:38:12.882133 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:12.882100    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:38:32.678561 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:32.678521    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:38:41.474111 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:41.474073    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:38:50.776952 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:50.776920    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:38:51.190006 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.189965    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"]
Sep 11 01:38:51.190391 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.190375    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy"
Sep 11 01:38:51.190496 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.190393    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef7c2c7-9857-4238-94a5-bb1ed0d5420e" containerName="istio-proxy"
Sep 11 01:38:51.190496 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.190421    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy"
Sep 11 01:38:51.190496 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.190431    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="421d2f92-ddcf-4b84-ba3e-fc605947f26c" containerName="istio-proxy"
Sep 11 01:38:51.193398 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.193378    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.195802 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.195783    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fdvhg\""
Sep 11 01:38:51.195915 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.195844    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 01:38:51.197002 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.196981    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 01:38:51.197066 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.197016    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 01:38:51.197066 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.197049    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 01:38:51.203128 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.203107    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"]
Sep 11 01:38:51.257298 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.257266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-maas-api-tls\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.257445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.257322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-openshift-service-ca\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.257445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.257348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7w4h\" (UniqueName: \"kubernetes.io/projected/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-kube-api-access-r7w4h\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.257445 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.257427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-metrics-certs\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.345476 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.345445    2583 scope.go:117] "RemoveContainer" containerID="dc648402e7be9591acb2c130f70b839ed7d7c830d6fbbef0a472a108844dd631"
Sep 11 01:38:51.358689 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.358667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-metrics-certs\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.358757 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.358722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-maas-api-tls\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.358802 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.358753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-openshift-service-ca\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.358802 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.358786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7w4h\" (UniqueName: \"kubernetes.io/projected/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-kube-api-access-r7w4h\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.359476 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.359454    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-openshift-service-ca\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.361338 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.361293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-maas-api-tls\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.361418 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.361293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-metrics-certs\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.367378 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.367360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7w4h\" (UniqueName: \"kubernetes.io/projected/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-kube-api-access-r7w4h\") pod \"maas-api-7b7ff8bc87-smf5x\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.504383 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.504294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:51.778036 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.778010    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"]
Sep 11 01:38:51.780110 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:38:51.780082    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecdf0c5_61a2_4ead_9f27_8ae4b1e5b956.slice/crio-8bed2fe6a22a3ed9c14131f2bc5299dc25073432ffb5203656feae0ad2bbf430 WatchSource:0}: Error finding container 8bed2fe6a22a3ed9c14131f2bc5299dc25073432ffb5203656feae0ad2bbf430: Status 404 returned error can't find the container with id 8bed2fe6a22a3ed9c14131f2bc5299dc25073432ffb5203656feae0ad2bbf430
Sep 11 01:38:51.781642 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:51.781621    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 01:38:52.468206 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:52.468117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x" event={"ID":"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956","Type":"ContainerStarted","Data":"625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884"}
Sep 11 01:38:52.468206 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:52.468165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x" event={"ID":"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956","Type":"ContainerStarted","Data":"8bed2fe6a22a3ed9c14131f2bc5299dc25073432ffb5203656feae0ad2bbf430"}
Sep 11 01:38:52.468206 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:52.468193    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:52.483100 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:52.483048    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x" podStartSLOduration=1.103989048 podStartE2EDuration="1.48303017s" totalImagesPullingTime="379.041122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:38:52.482811514 +0000 UTC m=+2462.019462717" watchObservedRunningTime="2026-09-11 01:38:52.48303017 +0000 UTC m=+2462.019681368"
Sep 11 01:38:58.478500 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:58.478469    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:38:58.886186 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:58.886154    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:38:58.917895 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:58.917858    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"]
Sep 11 01:38:58.921082 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:58.921061    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:58.931266 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:58.931243    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"]
Sep 11 01:38:59.024249 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.024216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdrs\" (UniqueName: \"kubernetes.io/projected/a5a4ca9d-d223-46bf-9129-01f515045f08-kube-api-access-8sdrs\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.024430 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.024265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a4ca9d-d223-46bf-9129-01f515045f08-maas-api-tls\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.024478 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.024420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a4ca9d-d223-46bf-9129-01f515045f08-metrics-certs\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.024478 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.024462    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a4ca9d-d223-46bf-9129-01f515045f08-openshift-service-ca\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.125118 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.125088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a4ca9d-d223-46bf-9129-01f515045f08-metrics-certs\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.125118 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.125122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a4ca9d-d223-46bf-9129-01f515045f08-openshift-service-ca\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.125420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.125157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdrs\" (UniqueName: \"kubernetes.io/projected/a5a4ca9d-d223-46bf-9129-01f515045f08-kube-api-access-8sdrs\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.125420 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.125335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a4ca9d-d223-46bf-9129-01f515045f08-maas-api-tls\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.126029 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.126003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a4ca9d-d223-46bf-9129-01f515045f08-openshift-service-ca\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.127627 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.127607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a4ca9d-d223-46bf-9129-01f515045f08-maas-api-tls\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.127717 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.127665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a4ca9d-d223-46bf-9129-01f515045f08-metrics-certs\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.132574 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.132555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdrs\" (UniqueName: \"kubernetes.io/projected/a5a4ca9d-d223-46bf-9129-01f515045f08-kube-api-access-8sdrs\") pod \"maas-api-fcdb7dc4c-d5m7p\" (UID: \"a5a4ca9d-d223-46bf-9129-01f515045f08\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.234591 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.234492    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:38:59.507102 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:38:59.507062    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"]
Sep 11 01:38:59.508755 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:38:59.508725    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a4ca9d_d223_46bf_9129_01f515045f08.slice/crio-552b43803150dea92cb962468e8781cd1d019c25e8d87a14818aceeddd6f82e8 WatchSource:0}: Error finding container 552b43803150dea92cb962468e8781cd1d019c25e8d87a14818aceeddd6f82e8: Status 404 returned error can't find the container with id 552b43803150dea92cb962468e8781cd1d019c25e8d87a14818aceeddd6f82e8
Sep 11 01:39:00.506567 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:00.506532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p" event={"ID":"a5a4ca9d-d223-46bf-9129-01f515045f08","Type":"ContainerStarted","Data":"79d542d8aee72cffc8a109736f8996a69af121fff84374a73936ab84950da109"}
Sep 11 01:39:00.506717 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:00.506581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p" event={"ID":"a5a4ca9d-d223-46bf-9129-01f515045f08","Type":"ContainerStarted","Data":"552b43803150dea92cb962468e8781cd1d019c25e8d87a14818aceeddd6f82e8"}
Sep 11 01:39:00.506717 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:00.506656    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:39:00.525125 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:00.525022    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p" podStartSLOduration=1.760560038 podStartE2EDuration="2.525000216s" totalImagesPullingTime="764.440178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:39:00.522620385 +0000 UTC m=+2470.059271591" watchObservedRunningTime="2026-09-11 01:39:00.525000216 +0000 UTC m=+2470.061651419"
Sep 11 01:39:05.994580 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:05.994548    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:39:06.517039 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.517012    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-d5m7p"
Sep 11 01:39:06.562552 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.562513    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"]
Sep 11 01:39:06.562829 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.562773    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x" podUID="aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" containerName="maas-api" containerID="cri-o://625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884" gracePeriod=30
Sep 11 01:39:06.812842 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.812820    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:39:06.893624 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.893596    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7w4h\" (UniqueName: \"kubernetes.io/projected/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-kube-api-access-r7w4h\") pod \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") "
Sep 11 01:39:06.893815 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.893696    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-metrics-certs\") pod \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") "
Sep 11 01:39:06.893815 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.893744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-openshift-service-ca\") pod \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") "
Sep 11 01:39:06.893815 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.893805    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-maas-api-tls\") pod \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\" (UID: \"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956\") "
Sep 11 01:39:06.894183 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.894147    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" (UID: "aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 01:39:06.895751 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.895723    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" (UID: "aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:39:06.895856 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.895768    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-kube-api-access-r7w4h" (OuterVolumeSpecName: "kube-api-access-r7w4h") pod "aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" (UID: "aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956"). InnerVolumeSpecName "kube-api-access-r7w4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 01:39:06.895856 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.895820    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" (UID: "aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 01:39:06.995134 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.995100    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-metrics-certs\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:39:06.995134 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.995129    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-openshift-service-ca\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:39:06.995134 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.995142    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-maas-api-tls\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:39:06.995593 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:06.995154    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7w4h\" (UniqueName: \"kubernetes.io/projected/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956-kube-api-access-r7w4h\") on node \"ip-10-0-142-82.ec2.internal\" DevicePath \"\""
Sep 11 01:39:07.543902 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.543866    2583 generic.go:356] "Generic (PLEG): container finished" podID="aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" containerID="625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884" exitCode=0
Sep 11 01:39:07.544108 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.543912    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"
Sep 11 01:39:07.544108 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.543942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x" event={"ID":"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956","Type":"ContainerDied","Data":"625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884"}
Sep 11 01:39:07.544108 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.544005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x" event={"ID":"aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956","Type":"ContainerDied","Data":"8bed2fe6a22a3ed9c14131f2bc5299dc25073432ffb5203656feae0ad2bbf430"}
Sep 11 01:39:07.544108 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.544029    2583 scope.go:117] "RemoveContainer" containerID="625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884"
Sep 11 01:39:07.554960 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.554937    2583 scope.go:117] "RemoveContainer" containerID="625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884"
Sep 11 01:39:07.555353 ip-10-0-142-82 kubenswrapper[2583]: E0911 01:39:07.555332    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884\": container with ID starting with 625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884 not found: ID does not exist" containerID="625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884"
Sep 11 01:39:07.555436 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.555362    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884"} err="failed to get container status \"625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884\": rpc error: code = NotFound desc = could not find container \"625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884\": container with ID starting with 625abaa875e3d42970ea0a976c83f1f06d5f6dfbd89cc343466ece8b8f480884 not found: ID does not exist"
Sep 11 01:39:07.559297 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.559273    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"]
Sep 11 01:39:07.563273 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:07.563255    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-smf5x"]
Sep 11 01:39:09.038415 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:09.038384    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" path="/var/lib/kubelet/pods/aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956/volumes"
Sep 11 01:39:11.578296 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:11.578265    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:39:18.279838 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:18.279805    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qzt5d"]
Sep 11 01:39:27.242366 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:27.242336    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bz9m2_d283a886-ef93-4943-b2e2-650c58a45ae0/manager/0.log"
Sep 11 01:39:27.488719 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:27.488688    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7z9xk_f67d2b7e-3262-4cca-8b5f-538ce246dc6e/server/0.log"
Sep 11 01:39:27.606921 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:27.606892    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rz5pw_5dd9fca6-85c2-434d-a586-8c0c68bd5143/manager/0.log"
Sep 11 01:39:27.719732 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:27.719705    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8bd47c7d5-jmgzc_dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7/copy-manifests/0.log"
Sep 11 01:39:27.731404 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:27.731375    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8bd47c7d5-jmgzc_dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7/manager/0.log"
Sep 11 01:39:28.334294 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:28.334264    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fcdb7dc4c-d5m7p_a5a4ca9d-d223-46bf-9129-01f515045f08/maas-api/0.log"
Sep 11 01:39:28.559715 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:28.559686    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818155-6lft8_77b6112d-a4d8-4d40-93d3-b8bf0773cb4f/cleanup/0.log"
Sep 11 01:39:30.096245 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:30.096216    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2f8dz_d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5/manager/0.log"
Sep 11 01:39:30.261283 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:30.261258    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-qzt5d_be7c29fc-a1b6-415a-8e7a-cb28876bc726/limitador/0.log"
Sep 11 01:39:31.600221 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:31.600191    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6994df4f88-cwhzx_dfdd4c42-aa61-4c97-92fd-7cdca51780bd/router/0.log"
Sep 11 01:39:32.431985 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:32.431953    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnspq7_7e29af54-d922-4450-9098-1369a671387b/main/0.log"
Sep 11 01:39:32.544484 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:32.544455    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vq56h_888d9584-591d-43bc-b31d-50580a2f1b27/main/0.log"
Sep 11 01:39:32.653439 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:32.653409    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-9jgdp_037fd866-ebac-4e5a-a139-8790d0282f6c/main/0.log"
Sep 11 01:39:40.976163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:40.976132    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m4sbv_a9227d14-ae58-49a3-9795-626c24256f78/global-pull-secret-syncer/0.log"
Sep 11 01:39:41.061539 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:41.061505    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qqpsr_e47f1012-5a54-441e-8fe1-f12d68b88038/konnectivity-agent/0.log"
Sep 11 01:39:41.164622 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:41.164584    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-82.ec2.internal_f1dfb5bbad91f5e0587cf031dfbabf92/haproxy/0.log"
Sep 11 01:39:45.873548 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:45.873462    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2f8dz_d482aff3-78b1-4d5b-aa1d-417c4a9fb8c5/manager/0.log"
Sep 11 01:39:45.900654 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:45.900632    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-qzt5d_be7c29fc-a1b6-415a-8e7a-cb28876bc726/limitador/0.log"
Sep 11 01:39:47.909878 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:47.909844    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-qjmm2_bde7aa89-7b63-4dfb-b48e-12ea9734ab7d/cluster-monitoring-operator/0.log"
Sep 11 01:39:48.289172 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:48.289136    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nr6xd_30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5/node-exporter/0.log"
Sep 11 01:39:48.311157 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:48.311135    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nr6xd_30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5/kube-rbac-proxy/0.log"
Sep 11 01:39:48.331294 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:48.331275    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nr6xd_30c95fb1-24c5-4f71-8ff7-3fb85b2d9ec5/init-textfile/0.log"
Sep 11 01:39:50.371374 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:50.371329    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-c2z44_614f95d1-510e-4863-ac36-e2aa7a01ccef/networking-console-plugin/0.log"
Sep 11 01:39:50.954165 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:50.954114    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/1.log"
Sep 11 01:39:50.959000 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:50.958976    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mrjfb_018f8615-58a3-4dc7-bc26-f5d740868701/console-operator/2.log"
Sep 11 01:39:51.501438 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.501405    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"]
Sep 11 01:39:51.501772 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.501735    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" containerName="maas-api"
Sep 11 01:39:51.501772 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.501745    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" containerName="maas-api"
Sep 11 01:39:51.501844 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.501814    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aecdf0c5-61a2-4ead-9f27-8ae4b1e5b956" containerName="maas-api"
Sep 11 01:39:51.506088 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.506069    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.508292 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.508272    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjvqr\"/\"kube-root-ca.crt\""
Sep 11 01:39:51.509038 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.509022    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjvqr\"/\"openshift-service-ca.crt\""
Sep 11 01:39:51.509131 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.509038    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sjvqr\"/\"default-dockercfg-dv4qx\""
Sep 11 01:39:51.513907 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.513886    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"]
Sep 11 01:39:51.571933 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.571908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-podres\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.572109 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.571991    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-lib-modules\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.572109 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.572010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2rv\" (UniqueName: \"kubernetes.io/projected/c9ff8ea1-3d67-48c4-8858-03504a443221-kube-api-access-2h2rv\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.572109 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.572039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-sys\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.572109 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.572078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-proc\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673468 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-podres\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673468 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-lib-modules\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2rv\" (UniqueName: \"kubernetes.io/projected/c9ff8ea1-3d67-48c4-8858-03504a443221-kube-api-access-2h2rv\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-sys\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-proc\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673562    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-podres\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-proc\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-sys\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.673667 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.673647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9ff8ea1-3d67-48c4-8858-03504a443221-lib-modules\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.681369 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.681352    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2rv\" (UniqueName: \"kubernetes.io/projected/c9ff8ea1-3d67-48c4-8858-03504a443221-kube-api-access-2h2rv\") pod \"perf-node-gather-daemonset-8jv24\" (UID: \"c9ff8ea1-3d67-48c4-8858-03504a443221\") " pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:51.816163 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:51.816124    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:52.084132 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:52.084106    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"]
Sep 11 01:39:52.086503 ip-10-0-142-82 kubenswrapper[2583]: W0911 01:39:52.086480    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc9ff8ea1_3d67_48c4_8858_03504a443221.slice/crio-95df9071b527a18f35dde6381c672f61dab913b7d45cc982a26abb37b327215d WatchSource:0}: Error finding container 95df9071b527a18f35dde6381c672f61dab913b7d45cc982a26abb37b327215d: Status 404 returned error can't find the container with id 95df9071b527a18f35dde6381c672f61dab913b7d45cc982a26abb37b327215d
Sep 11 01:39:52.253925 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:52.253903    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-qvzgd_31932c40-b8a9-4ab3-8901-6d3d970aa7c5/volume-data-source-validator/0.log"
Sep 11 01:39:52.752257 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:52.752225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24" event={"ID":"c9ff8ea1-3d67-48c4-8858-03504a443221","Type":"ContainerStarted","Data":"caacea2d3683ddeac24adf96834d2c4468db86fa561023433ff25817744a7c55"}
Sep 11 01:39:52.752665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:52.752274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24" event={"ID":"c9ff8ea1-3d67-48c4-8858-03504a443221","Type":"ContainerStarted","Data":"95df9071b527a18f35dde6381c672f61dab913b7d45cc982a26abb37b327215d"}
Sep 11 01:39:52.752665 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:52.752355    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:52.768486 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:52.768442    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24" podStartSLOduration=1.768426115 podStartE2EDuration="1.768426115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 01:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:39:52.767409086 +0000 UTC m=+2522.304060287" watchObservedRunningTime="2026-09-11 01:39:52.768426115 +0000 UTC m=+2522.305077316"
Sep 11 01:39:53.422708 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:53.422649    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rncnx_583bbbc2-81a6-4a5f-b224-46ec7e1ecd86/dns/0.log"
Sep 11 01:39:53.441727 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:53.441702    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rncnx_583bbbc2-81a6-4a5f-b224-46ec7e1ecd86/kube-rbac-proxy/0.log"
Sep 11 01:39:53.465264 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:53.465241    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7zc7_c41dccbf-cc64-4cbf-8d90-c270f6cd6f95/dns-node-resolver/0.log"
Sep 11 01:39:54.106040 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:54.106011    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-47kkt_9dda75f8-48cf-4a74-ba9a-e6df5c3b8e35/node-ca/0.log"
Sep 11 01:39:55.726772 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:55.726726    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6994df4f88-cwhzx_dfdd4c42-aa61-4c97-92fd-7cdca51780bd/router/0.log"
Sep 11 01:39:56.621053 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:56.621012    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drvkn_d23e8729-271c-4456-a421-4ef108717362/serve-healthcheck-canary/0.log"
Sep 11 01:39:57.072072 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:57.072032    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-km5j6_a10e86f3-6bcd-42da-a580-e218495a161b/insights-operator/0.log"
Sep 11 01:39:57.072488 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:57.072470    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-km5j6_a10e86f3-6bcd-42da-a580-e218495a161b/insights-operator/1.log"
Sep 11 01:39:57.157086 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:57.157064    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvmm6_1c50397a-edd8-48e0-87ba-71e8f359f890/kube-rbac-proxy/0.log"
Sep 11 01:39:57.175889 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:57.175865    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvmm6_1c50397a-edd8-48e0-87ba-71e8f359f890/exporter/0.log"
Sep 11 01:39:57.197166 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:57.197140    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvmm6_1c50397a-edd8-48e0-87ba-71e8f359f890/extractor/0.log"
Sep 11 01:39:58.766783 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:58.766755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sjvqr/perf-node-gather-daemonset-8jv24"
Sep 11 01:39:59.524075 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:59.524041    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bz9m2_d283a886-ef93-4943-b2e2-650c58a45ae0/manager/0.log"
Sep 11 01:39:59.613865 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:59.613837    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7z9xk_f67d2b7e-3262-4cca-8b5f-538ce246dc6e/server/0.log"
Sep 11 01:39:59.692711 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:59.692681    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rz5pw_5dd9fca6-85c2-434d-a586-8c0c68bd5143/manager/0.log"
Sep 11 01:39:59.767999 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:59.767970    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8bd47c7d5-jmgzc_dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7/manager/0.log"
Sep 11 01:39:59.783407 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:39:59.783335    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8bd47c7d5-jmgzc_dd24253c-0f5f-4c69-a4a2-ce3f9e18f4f7/copy-manifests/0.log"
Sep 11 01:40:01.363551 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:01.363526    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bdf494464-gld47_d7555119-b990-429c-a16e-051c6f30b097/manager/0.log"
Sep 11 01:40:02.712710 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:02.712677    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6994df4f88-cwhzx_dfdd4c42-aa61-4c97-92fd-7cdca51780bd/router/0.log"
Sep 11 01:40:05.053703 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:05.053674    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6994df4f88-cwhzx_dfdd4c42-aa61-4c97-92fd-7cdca51780bd/router/0.log"
Sep 11 01:40:06.769425 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:06.769393    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-k9cb4_cefebf34-9896-4f29-84b3-596e6bbb0b9e/kube-storage-version-migrator-operator/1.log"
Sep 11 01:40:06.770260 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:06.770241    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-k9cb4_cefebf34-9896-4f29-84b3-596e6bbb0b9e/kube-storage-version-migrator-operator/0.log"
Sep 11 01:40:07.757219 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:07.757189    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rb67_68c89915-cc67-481d-bf53-0a2656dd28bc/kube-multus-additional-cni-plugins/0.log"
Sep 11 01:40:07.793079 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:07.793052    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rb67_68c89915-cc67-481d-bf53-0a2656dd28bc/egress-router-binary-copy/0.log"
Sep 11 01:40:07.826388 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:07.826214    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rb67_68c89915-cc67-481d-bf53-0a2656dd28bc/cni-plugins/0.log"
Sep 11 01:40:07.862053 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:07.862030    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rb67_68c89915-cc67-481d-bf53-0a2656dd28bc/bond-cni-plugin/0.log"
Sep 11 01:40:07.894888 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:07.894870    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rb67_68c89915-cc67-481d-bf53-0a2656dd28bc/routeoverride-cni/0.log"
Sep 11 01:40:07.928999 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:07.928982    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rb67_68c89915-cc67-481d-bf53-0a2656dd28bc/whereabouts-cni-bincopy/0.log"
Sep 11 01:40:07.964899 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:07.964881    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6rb67_68c89915-cc67-481d-bf53-0a2656dd28bc/whereabouts-cni/0.log"
Sep 11 01:40:08.361215 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:08.361186    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fjfls_45cbdc31-050e-4c41-baa8-0f184ca05be4/kube-multus/0.log"
Sep 11 01:40:08.534655 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:08.534630    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qrwkr_2fca14ea-d910-47a0-9563-3ea2ff660cb6/network-metrics-daemon/0.log"
Sep 11 01:40:08.551558 ip-10-0-142-82 kubenswrapper[2583]: I0911 01:40:08.551533    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qrwkr_2fca14ea-d910-47a0-9563-3ea2ff660cb6/kube-rbac-proxy/0.log"